[
  {
    "path": "LICENSE",
    "content": "GNU GENERAL PUBLIC LICENSE\n                       Version 2, June 1991\n\n Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>\n 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA\n Everyone is permitted to copy and distribute verbatim copies\n of this license document, but changing it is not allowed.\n\n                            Preamble\n\n  The licenses for most software are designed to take away your\nfreedom to share and change it.  By contrast, the GNU General Public\nLicense is intended to guarantee your freedom to share and change free\nsoftware--to make sure the software is free for all its users.  This\nGeneral Public License applies to most of the Free Software\nFoundation's software and to any other program whose authors commit to\nusing it.  (Some other Free Software Foundation software is covered by\nthe GNU Lesser General Public License instead.)  You can apply it to\nyour programs, too.\n\n  When we speak of free software, we are referring to freedom, not\nprice.  Our General Public Licenses are designed to make sure that you\nhave the freedom to distribute copies of free software (and charge for\nthis service if you wish), that you receive source code or can get it\nif you want it, that you can change the software or use pieces of it\nin new free programs; and that you know you can do these things.\n\n  To protect your rights, we need to make restrictions that forbid\nanyone to deny you these rights or to ask you to surrender the rights.\nThese restrictions translate to certain responsibilities for you if you\ndistribute copies of the software, or if you modify it.\n\n  For example, if you distribute copies of such a program, whether\ngratis or for a fee, you must give the recipients all the rights that\nyou have.  You must make sure that they, too, receive or can get the\nsource code.  And you must show them these terms so they know their\nrights.\n\n  We protect your rights with two steps: (1) copyright the software, and\n(2) offer you this license which gives you legal permission to copy,\ndistribute and/or modify the software.\n\n  Also, for each author's protection and ours, we want to make certain\nthat everyone understands that there is no warranty for this free\nsoftware.  If the software is modified by someone else and passed on, we\nwant its recipients to know that what they have is not the original, so\nthat any problems introduced by others will not reflect on the original\nauthors' reputations.\n\n  Finally, any free program is threatened constantly by software\npatents.  We wish to avoid the danger that redistributors of a free\nprogram will individually obtain patent licenses, in effect making the\nprogram proprietary.  To prevent this, we have made it clear that any\npatent must be licensed for everyone's free use or not licensed at all.\n\n  The precise terms and conditions for copying, distribution and\nmodification follow.\n\n                    GNU GENERAL PUBLIC LICENSE\n   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION\n\n  0. This License applies to any program or other work which contains\na notice placed by the copyright holder saying it may be distributed\nunder the terms of this General Public License.  The \"Program\", below,\nrefers to any such program or work, and a \"work based on the Program\"\nmeans either the Program or any derivative work under copyright law:\nthat is to say, a work containing the Program or a portion of it,\neither verbatim or with modifications and/or translated into another\nlanguage.  (Hereinafter, translation is included without limitation in\nthe term \"modification\".)  Each licensee is addressed as \"you\".\n\nActivities other than copying, distribution and modification are not\ncovered by this License; they are outside its scope.  The act of\nrunning the Program is not restricted, and the output from the Program\nis covered only if its contents constitute a work based on the\nProgram (independent of having been made by running the Program).\nWhether that is true depends on what the Program does.\n\n  1. You may copy and distribute verbatim copies of the Program's\nsource code as you receive it, in any medium, provided that you\nconspicuously and appropriately publish on each copy an appropriate\ncopyright notice and disclaimer of warranty; keep intact all the\nnotices that refer to this License and to the absence of any warranty;\nand give any other recipients of the Program a copy of this License\nalong with the Program.\n\nYou may charge a fee for the physical act of transferring a copy, and\nyou may at your option offer warranty protection in exchange for a fee.\n\n  2. You may modify your copy or copies of the Program or any portion\nof it, thus forming a work based on the Program, and copy and\ndistribute such modifications or work under the terms of Section 1\nabove, provided that you also meet all of these conditions:\n\n    a) You must cause the modified files to carry prominent notices\n    stating that you changed the files and the date of any change.\n\n    b) You must cause any work that you distribute or publish, that in\n    whole or in part contains or is derived from the Program or any\n    part thereof, to be licensed as a whole at no charge to all third\n    parties under the terms of this License.\n\n    c) If the modified program normally reads commands interactively\n    when run, you must cause it, when started running for such\n    interactive use in the most ordinary way, to print or display an\n    announcement including an appropriate copyright notice and a\n    notice that there is no warranty (or else, saying that you provide\n    a warranty) and that users may redistribute the program under\n    these conditions, and telling the user how to view a copy of this\n    License.  (Exception: if the Program itself is interactive but\n    does not normally print such an announcement, your work based on\n    the Program is not required to print an announcement.)\n\nThese requirements apply to the modified work as a whole.  If\nidentifiable sections of that work are not derived from the Program,\nand can be reasonably considered independent and separate works in\nthemselves, then this License, and its terms, do not apply to those\nsections when you distribute them as separate works.  But when you\ndistribute the same sections as part of a whole which is a work based\non the Program, the distribution of the whole must be on the terms of\nthis License, whose permissions for other licensees extend to the\nentire whole, and thus to each and every part regardless of who wrote it.\n\nThus, it is not the intent of this section to claim rights or contest\nyour rights to work written entirely by you; rather, the intent is to\nexercise the right to control the distribution of derivative or\ncollective works based on the Program.\n\nIn addition, mere aggregation of another work not based on the Program\nwith the Program (or with a work based on the Program) on a volume of\na storage or distribution medium does not bring the other work under\nthe scope of this License.\n\n  3. You may copy and distribute the Program (or a work based on it,\nunder Section 2) in object code or executable form under the terms of\nSections 1 and 2 above provided that you also do one of the following:\n\n    a) Accompany it with the complete corresponding machine-readable\n    source code, which must be distributed under the terms of Sections\n    1 and 2 above on a medium customarily used for software interchange; or,\n\n    b) Accompany it with a written offer, valid for at least three\n    years, to give any third party, for a charge no more than your\n    cost of physically performing source distribution, a complete\n    machine-readable copy of the corresponding source code, to be\n    distributed under the terms of Sections 1 and 2 above on a medium\n    customarily used for software interchange; or,\n\n    c) Accompany it with the information you received as to the offer\n    to distribute corresponding source code.  (This alternative is\n    allowed only for noncommercial distribution and only if you\n    received the program in object code or executable form with such\n    an offer, in accord with Subsection b above.)\n\nThe source code for a work means the preferred form of the work for\nmaking modifications to it.  For an executable work, complete source\ncode means all the source code for all modules it contains, plus any\nassociated interface definition files, plus the scripts used to\ncontrol compilation and installation of the executable.  However, as a\nspecial exception, the source code distributed need not include\nanything that is normally distributed (in either source or binary\nform) with the major components (compiler, kernel, and so on) of the\noperating system on which the executable runs, unless that component\nitself accompanies the executable.\n\nIf distribution of executable or object code is made by offering\naccess to copy from a designated place, then offering equivalent\naccess to copy the source code from the same place counts as\ndistribution of the source code, even though third parties are not\ncompelled to copy the source along with the object code.\n\n  4. You may not copy, modify, sublicense, or distribute the Program\nexcept as expressly provided under this License.  Any attempt\notherwise to copy, modify, sublicense or distribute the Program is\nvoid, and will automatically terminate your rights under this License.\nHowever, parties who have received copies, or rights, from you under\nthis License will not have their licenses terminated so long as such\nparties remain in full compliance.\n\n  5. You are not required to accept this License, since you have not\nsigned it.  However, nothing else grants you permission to modify or\ndistribute the Program or its derivative works.  These actions are\nprohibited by law if you do not accept this License.  Therefore, by\nmodifying or distributing the Program (or any work based on the\nProgram), you indicate your acceptance of this License to do so, and\nall its terms and conditions for copying, distributing or modifying\nthe Program or works based on it.\n\n  6. Each time you redistribute the Program (or any work based on the\nProgram), the recipient automatically receives a license from the\noriginal licensor to copy, distribute or modify the Program subject to\nthese terms and conditions.  You may not impose any further\nrestrictions on the recipients' exercise of the rights granted herein.\nYou are not responsible for enforcing compliance by third parties to\nthis License.\n\n  7. If, as a consequence of a court judgment or allegation of patent\ninfringement or for any other reason (not limited to patent issues),\nconditions are imposed on you (whether by court order, agreement or\notherwise) that contradict the conditions of this License, they do not\nexcuse you from the conditions of this License.  If you cannot\ndistribute so as to satisfy simultaneously your obligations under this\nLicense and any other pertinent obligations, then as a consequence you\nmay not distribute the Program at all.  For example, if a patent\nlicense would not permit royalty-free redistribution of the Program by\nall those who receive copies directly or indirectly through you, then\nthe only way you could satisfy both it and this License would be to\nrefrain entirely from distribution of the Program.\n\nIf any portion of this section is held invalid or unenforceable under\nany particular circumstance, the balance of the section is intended to\napply and the section as a whole is intended to apply in other\ncircumstances.\n\nIt is not the purpose of this section to induce you to infringe any\npatents or other property right claims or to contest validity of any\nsuch claims; this section has the sole purpose of protecting the\nintegrity of the free software distribution system, which is\nimplemented by public license practices.  Many people have made\ngenerous contributions to the wide range of software distributed\nthrough that system in reliance on consistent application of that\nsystem; it is up to the author/donor to decide if he or she is willing\nto distribute software through any other system and a licensee cannot\nimpose that choice.\n\nThis section is intended to make thoroughly clear what is believed to\nbe a consequence of the rest of this License.\n\n  8. If the distribution and/or use of the Program is restricted in\ncertain countries either by patents or by copyrighted interfaces, the\noriginal copyright holder who places the Program under this License\nmay add an explicit geographical distribution limitation excluding\nthose countries, so that distribution is permitted only in or among\ncountries not thus excluded.  In such case, this License incorporates\nthe limitation as if written in the body of this License.\n\n  9. The Free Software Foundation may publish revised and/or new versions\nof the General Public License from time to time.  Such new versions will\nbe similar in spirit to the present version, but may differ in detail to\naddress new problems or concerns.\n\nEach version is given a distinguishing version number.  If the Program\nspecifies a version number of this License which applies to it and \"any\nlater version\", you have the option of following the terms and conditions\neither of that version or of any later version published by the Free\nSoftware Foundation.  If the Program does not specify a version number of\nthis License, you may choose any version ever published by the Free Software\nFoundation.\n\n  10. If you wish to incorporate parts of the Program into other free\nprograms whose distribution conditions are different, write to the author\nto ask for permission.  For software which is copyrighted by the Free\nSoftware Foundation, write to the Free Software Foundation; we sometimes\nmake exceptions for this.  Our decision will be guided by the two goals\nof preserving the free status of all derivatives of our free software and\nof promoting the sharing and reuse of software generally.\n\n                            NO WARRANTY\n\n  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY\nFOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN\nOTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES\nPROVIDE THE PROGRAM \"AS IS\" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED\nOR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF\nMERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS\nTO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE\nPROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,\nREPAIR OR CORRECTION.\n\n  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING\nWILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR\nREDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,\nINCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING\nOUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED\nTO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY\nYOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER\nPROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE\nPOSSIBILITY OF SUCH DAMAGES.\n\n                     END OF TERMS AND CONDITIONS\n\n            How to Apply These Terms to Your New Programs\n\n  If you develop a new program, and you want it to be of the greatest\npossible use to the public, the best way to achieve this is to make it\nfree software which everyone can redistribute and change under these terms.\n\n  To do so, attach the following notices to the program.  It is safest\nto attach them to the start of each source file to most effectively\nconvey the exclusion of warranty; and each file should have at least\nthe \"copyright\" line and a pointer to where the full notice is found.\n\n    {description}\n    Copyright (C) {year}  {fullname}\n\n    This program is free software; you can redistribute it and/or modify\n    it under the terms of the GNU General Public License as published by\n    the Free Software Foundation; either version 2 of the License, or\n    (at your option) any later version.\n\n    This program is distributed in the hope that it will be useful,\n    but WITHOUT ANY WARRANTY; without even the implied warranty of\n    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n    GNU General Public License for more details.\n\n    You should have received a copy of the GNU General Public License along\n    with this program; if not, write to the Free Software Foundation, Inc.,\n    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.\n\nAlso add information on how to contact you by electronic and paper mail.\n\nIf the program is interactive, make it output a short notice like this\nwhen it starts in an interactive mode:\n\n    Gnomovision version 69, Copyright (C) year name of author\n    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.\n    This is free software, and you are welcome to redistribute it\n    under certain conditions; type `show c' for details.\n\nThe hypothetical commands `show w' and `show c' should show the appropriate\nparts of the General Public License.  Of course, the commands you use may\nbe called something other than `show w' and `show c'; they could even be\nmouse-clicks or menu items--whatever suits your program.\n\nYou should also get your employer (if you work as a programmer) or your\nschool, if any, to sign a \"copyright disclaimer\" for the program, if\nnecessary.  Here is a sample; alter the names:\n\n  Yoyodyne, Inc., hereby disclaims all copyright interest in the program\n  `Gnomovision' (which makes passes at compilers) written by James Hacker.\n\n  {signature of Ty Coon}, 1 April 1989\n  Ty Coon, President of Vice\n\nThis General Public License does not permit incorporating your program into\nproprietary programs.  If your program is a subroutine library, you may\nconsider it more useful to permit linking proprietary applications with the\nlibrary.  If this is what you want to do, use the GNU Lesser General\nPublic License instead of this License."
  },
  {
    "path": "README.md",
    "content": "perf-tools\n==========\n\nA miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events (aka the \"perf\" command). Both ftrace and perf are core Linux tracing tools, included in the kernel source. Your system probably has ftrace already, and perf is often just a package add (see Prerequisites).\n\nThese tools are designed to be easy to install (fewest dependencies), provide advanced performance observability, and be simple to use: do one thing and do it well. This collection was created by Brendan Gregg (author of the DTraceToolkit).\n\nMany of these tools employ workarounds so that functionality is possible on existing Linux kernels. Because of this, many tools have caveats (see man pages), and their implementation should be considered a placeholder until future kernel features, or new tracing subsystems, are added.\n\nThese are intended for Linux 3.2 and newer kernels. For Linux 2.6.x, see Warnings.\n\n## Presentation\n\nThese tools were introduced in the USENIX LISA 2014 presentation: Linux Performance Analysis: New Tools and Old Secrets\n\n- slides: http://www.slideshare.net/brendangregg/linux-performance-analysis-new-tools-and-old-secrets\n- video: https://www.usenix.org/conference/lisa14/conference-program/presentation/gregg\n\n## Contents\n<center><a href=\"images/perf-tools_2016.png\"><img src=\"images/perf-tools_2016.png\" border=0 width=700></a></center>\n\nUsing ftrace:\n\n- [iosnoop](iosnoop): trace disk I/O with details including latency. [Examples](examples/iosnoop_example.txt).\n- [iolatency](iolatency): summarize disk I/O latency as a histogram. [Examples](examples/iolatency_example.txt).\n- [execsnoop](execsnoop): trace process exec() with command line argument details. [Examples](examples/execsnoop_example.txt).\n- [opensnoop](opensnoop): trace open() syscalls showing filenames. [Examples](examples/opensnoop_example.txt).\n- [killsnoop](killsnoop): trace kill() signals showing process and signal details. [Examples](examples/killsnoop_example.txt).\n- fs/[cachestat](fs/cachestat): basic cache hit/miss statistics for the Linux page cache. [Examples](examples/cachestat_example.txt).\n- net/[tcpretrans](net/tcpretrans): show TCP retransmits, with address and other details. [Examples](examples/tcpretrans_example.txt).\n- system/[tpoint](system/tpoint): trace a given tracepoint. [Examples](examples/tpoint_example.txt).\n- kernel/[funccount](kernel/funccount): count kernel function calls, matching a string with wildcards. [Examples](examples/funccount_example.txt).\n- kernel/[functrace](kernel/functrace): trace kernel function calls, matching a string with wildcards. [Examples](examples/functrace_example.txt).\n- kernel/[funcslower](kernel/funcslower): trace kernel functions slower than a threshold. [Examples](examples/funcslower_example.txt).\n- kernel/[funcgraph](kernel/funcgraph): trace a graph of kernel function calls, showing children and times. [Examples](examples/funcgraph_example.txt).\n- kernel/[kprobe](kernel/kprobe): dynamically trace a kernel function call or its return, with variables. [Examples](examples/kprobe_example.txt).\n- user/[uprobe](user/uprobe): dynamically trace a user-level function call or its return, with variables. [Examples](examples/uprobe_example.txt).\n- tools/[reset-ftrace](tools/reset-ftrace): reset ftrace state if needed. [Examples](examples/reset-ftrace_example.txt).\n\nUsing perf_events:\n\n- misc/[perf-stat-hist](misc/perf-stat-hist): power-of aggregations for tracepoint variables. [Examples](examples/perf-stat-hist_example.txt).\n- [syscount](syscount): count syscalls by syscall or process. [Examples](examples/syscount_example.txt).\n- disk/[bitesize](disk/bitesize): histogram summary of disk I/O size. [Examples](examples/bitesize_example.txt).\n\nUsing eBPF:\n\n- As a preview of things to come, see the bcc tracing [Tools section](https://github.com/iovisor/bcc/blob/master/README.md#tracing). These use [bcc](https://github.com/iovisor/bcc), a front end for using [eBPF](http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html). bcc+eBPF will allow some of these tools to be rewritten and improved, and additional tools to be created.\n\n## Screenshots\n\nShowing new processes and arguments:\n\n<pre># <b>./execsnoop</b> \nTracing exec()s. Ctrl-C to end.\n   PID   PPID ARGS\n 22898  22004 man ls\n 22905  22898 preconv -e UTF-8\n 22908  22898 pager -s\n 22907  22898 nroff -mandoc -rLL=164n -rLT=164n -Tutf8\n 22906  22898 tbl\n 22911  22910 locale charmap\n 22912  22907 groff -mtty-char -Tutf8 -mandoc -rLL=164n -rLT=164n\n 22913  22912 troff -mtty-char -mandoc -rLL=164n -rLT=164n -Tutf8\n 22914  22912 grotty\n</pre>\n\nMeasuring block device I/O latency from queue insert to completion:\n\n<pre># <b>./iolatency -Q</b>\nTracing block I/O. Output every 1 seconds. Ctrl-C to end.\n\n  &gt;=(ms) .. &lt;(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1913     |######################################|\n       1 -> 2       : 438      |#########                             |\n       2 -> 4       : 100      |##                                    |\n       4 -> 8       : 145      |###                                   |\n       8 -> 16      : 43       |#                                     |\n      16 -> 32      : 43       |#                                     |\n      32 -> 64      : 1        |#                                     |\n\n[...]\n</pre>\n\nTracing the block:block_rq_insert tracepoint, with kernel stack traces, and only for reads:\n\n<pre># <b>./tpoint -s block:block_rq_insert 'rwbs ~ \"*R*\"'</b>\n   cksum-11908 [000] d... 7269839.919098: block_rq_insert: 202,1 R 0 () 736560 + 136 [cksum]\n   cksum-11908 [000] d... 7269839.919107: <stack trace>\n => __elv_add_request\n => blk_flush_plug_list\n => blk_finish_plug\n => __do_page_cache_readahead\n => ondemand_readahead\n => page_cache_async_readahead\n => generic_file_read_iter\n => new_sync_read\n => vfs_read\n => SyS_read\n => system_call_fastpath\n\n[...]\n</pre>\n\nCount kernel function calls beginning with \"bio_\", summarize every second:\n\n<pre># <b>./funccount -i 1 'bio_*'</b>\nTracing \"bio_*\"... Ctrl-C to end.\n\nFUNC                              COUNT\nbio_attempt_back_merge               26\nbio_get_nr_vecs                     361\nbio_alloc                           536\nbio_alloc_bioset                    536\nbio_endio                           536\nbio_free                            536\nbio_fs_destructor                   536\nbio_init                            536\nbio_integrity_enabled               536\nbio_put                             729\nbio_add_page                       1004\n\n[...]\n</pre>\n\nThere are many more examples in the [examples](examples) directory. Also see the [man pages](man/man8).\n\n## Prerequisites\n\nThe intent is as few as possible. Eg, a Linux 3.2 server without debuginfo. See the tool man page for specifics.\n\n### ftrace\n\nFTRACE configured in the kernel. You may already have this configured and available in your kernel version, as FTRACE was first added in 2.6.27. This requires CONFIG_FTRACE and other FTRACE options depending on the tool. Some tools (eg, funccount) require CONFIG_FUNCTION_PROFILER.\n\n### perf_events\n\nRequires the \"perf\" command to be installed. This is in the linux-tools-common package. After installing that, perf may tell you to install an additional linux-tools package (linux-tools-_kernel_version_). perf can also be built under tools/perf in the kernel source. See [perf_events Prerequisites](http://www.brendangregg.com/perf.html#Prerequisites) for more details about getting perf_events to work fully.\n\n### debugfs\n\nRequires a kernel with CONFIG_DEBUG_FS option enabled. As with FTRACE, this may already be enabled (debugfs was added in 2.6.10-rc3). The debugfs also needs to be mounted:\n\n```\n# mount -t debugfs none /sys/kernel/debug\n```\n\n### awk\n\nMany of there scripts use awk, and will try to use either mawk or gawk depending on the desired behavior: mawk for buffered output (because of its speed), and gawk for synchronous output (as fflush() works, allowing more efficient grouping of writes).\n\n## Install\n\nThese are just scripts. Either grab everything:\n\n```\ngit clone --depth 1 https://github.com/brendangregg/perf-tools\n```\n\nOr use the raw links on github to download individual scripts. Eg:\n\n```\nwget https://raw.githubusercontent.com/brendangregg/perf-tools/master/iosnoop\n```\n\nThis preserves tabs (which copy-n-paste can mess up).\n\n## Warnings\n\nFtrace was first added to Linux 2.6.27, and perf_events to Linux 2.6.31. These early versions had kernel bugs, and lockups and panics have been reported on 2.6.32 series kernels. This includes CentOS 6.x. If you must analyze older kernels, these tools may only be useful in a fault-tolerant environment, such as a lab with simulated issues. These tools have been primarily developed on Linux 3.2 and later kernels.\n\nDepending on the tool, there may also be overhead incurred. See the next section.\n\n## Internals and Overhead\n\nperf_events is evolving. This collection began development circa Linux 3.16, with Linux 3.2 servers as the main target, at a time when perf_events lacks certain programmatic capabilities (eg, custom in-kernel aggregations). It's possible these will be added in a forthcoming kernel release. Until then, many of these tools employ workarounds, tricks, and hacks in order to work. Some of these tools pass event data to user space for post-processing, which costs much higher overhead than in-kernel aggregations. The overhead of each tool is described in its man page.\n\n__WARNING__: In _extreme_ cases, your target application may run 5x slower when using these tools. Depending on the tool and kernel version, there may also be the risk of kernel panics. Read the program header for warnings, and test before use.\n\nIf the overhead is a problem, these tools can be improved. If a tool doesn't already, it could be rewritten in C to use perf_events_open() and mmap() for the trace buffer. It could also implement frequency counts in C, and operate on mmap() directly, rather than using awk/Perl/Python. Additional improvements are possible for ftrace-based tools, such as use of snapshots and per-instance buffers.\n\nSome of these tools are intended as short-term workarounds until more kernel capabilities exist, at which point they can be substantially rewritten. Older versions of these tools will be kept in this repository, for older kernel versions.\n\nAs my main target is a fleet of Linux 3.2 servers that do not have debuginfo, these tools try not to require it. At times, this makes the tool more brittle than it needs to be, as I'm employing workarounds (that may be kernel version and platform specific) instead of using debuginfo information (which can be generic). See the man page for detailed prerequisites for each tool.\n\nI've tried to use perf_events (\"perf\") where possible, since that interface has been developed for multi-user use. For various reasons I've often needed to use ftrace instead. ftrace is surprisingly powerful (thanks Steven Rostedt!), and not all of its features are exposed via perf, or in common usage. This tool collection is in some ways a demonstration of hidden Linux features using ftrace.\n\nSince things are changing, it's very possible you may find some tools don't work on your Linux kernel version. Some expertise and assembly will be required to fix them.\n\n## Links\n\nA case study and summary:\n\n- 13 Aug 2014: http://lwn.net/Articles/608497 Ftrace: The hidden light switch\n\nRelated articles:\n\n- 28 Jun 2015: http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html\n- 31 Dec 2014: http://www.brendangregg.com/blog/2014-12-31/linux-page-cache-hit-ratio.html\n- 06 Sep 2014: http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp-retransmit-tracing.html\n- 28 Jul 2014: http://www.brendangregg.com/blog/2014-07-28/execsnoop-for-linux.html\n- 25 Jul 2014: http://www.brendangregg.com/blog/2014-07-25/opensnoop-for-linux.html\n- 23 Jul 2014: http://www.brendangregg.com/blog/2014-07-23/linux-iosnoop-latency-heat-maps.html\n- 16 Jul 2014: http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html\n- 10 Jul 2014: http://www.brendangregg.com/blog/2014-07-10/perf-hacktogram.html\n"
  },
  {
    "path": "deprecated/README.md",
    "content": "Deprecated versions of tools.\n"
  },
  {
    "path": "deprecated/execsnoop-proc",
    "content": "#!/usr/bin/perl\n#\n# execsnoop - trace process exec() with arguments. /proc version.\n#             Written using Linux ftrace.\n#\n# This shows the execution of new processes, especially short-lived ones that\n# can be missed by sampling tools such as top(1).\n#\n# USAGE: ./execsnoop [-h] [-n name]\n#\n# REQUIREMENTS: FTRACE CONFIG, sched:sched_process_exec tracepoint (you may\n# already have these on recent kernels), and Perl.\n#\n# This traces exec() from the fork()->exec() sequence, which means it won't\n# catch new processes that only fork(), and, it will catch processes that\n# re-exec. This instruments sched:sched_process_exec without buffering, and then\n# in user-space (this program) reads PPID and process arguments asynchronously\n# from /proc.\n#\n# If the process traced is very short-lived, this program may miss reading\n# arguments and PPID details. In that case, \"<?>\" and \"?\" will be printed\n# respectively. This program is best-effort, and should be improved in the\n# future when other kernel capabilities are made available. If you need a\n# more reliable tool now, then consider other tracing alternatives (eg,\n# SystemTap). This tool is really a proof of concept to see what ftrace can\n# currently do.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the execsnoop(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 07-Jul-2014\tBrendan Gregg\tCreated this.\n\nuse strict;\nuse warnings;\nuse POSIX qw(strftime);\nuse Getopt::Long;\nmy $tracing = \"/sys/kernel/debug/tracing\";\nmy $flock = \"/var/tmp/.ftrace-lock\";\nmy $tpdir = \"sched/sched_process_exec\";\nmy $tptext = $tpdir; $tptext =~ s/\\//:/;\nlocal $SIG{INT} = \\&cleanup;\nlocal $SIG{QUIT} = \\&cleanup;\nlocal $SIG{TERM} = \\&cleanup;\nlocal $SIG{PIPE} = \\&cleanup;\nlocal $SIG{HUP} = \\&cleanup;\n$| = 1;\n\n### options\nmy ($name, $help);\nGetOptions(\"name=s\" => \\$name,\n           \"help\"   => \\$help)\nor usage();\nusage() if $help;\nsub usage {\n\tprint STDERR \"USAGE: execsnoop [-h] [-n name]\\n\";\n\tprint STDERR \"   eg,\\n\";\n\tprint STDERR \"       execsnoop -n ls\t# show \\\"ls\\\" cmds only.\\n\";\n\texit;\n}\n\nsub ldie {\n\tunlink $flock;\n\tdie @_;\n}\n\nsub writeto {\n\tmy ($string, $file) = @_;\n\topen FILE, \">$file\" or return 0;\n\tprint FILE $string or return 0;\n\tclose FILE or return 0;\n}\n\n### check permissions\nchdir \"$tracing\" or ldie \"ERROR: accessing tracing. Root? Kernel has FTRACE?\" .\n    \"\\ndebugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\";\n\n### ftrace lock\nif (-e $flock) {\n\topen FLOCK, $flock; my $fpid = <FLOCK>; chomp $fpid; close FLOCK;\n\tdie \"ERROR: ftrace may be in use by PID $fpid ($flock)\";\n}\nwriteto \"$$\", $flock or die \"ERROR: unable to write $flock.\";\n\n### setup and begin tracing\nwriteto \"nop\", \"current_tracer\" or ldie \"ERROR: disabling current_tracer.\";\nwriteto \"1\", \"events/$tpdir/enable\" or ldie \"ERROR: enabling tracepoint \" .\n    \"\\\"$tptext\\\" (tracepoint missing in this kernel version?)\";\nopen TPIPE, \"trace_pipe\"  or warn \"ERROR: opening trace_pipe.\";\nprintf \"%-8s %6s %6s %s\\n\",  \"TIME\", \"PID\", \"PPID\", \"ARGS\";\n\nwhile (<TPIPE>) {\n\tmy ($taskpid, $rest) = split;\n\tmy ($task, $pid) = $taskpid =~ /(.*)-(\\d+)/;\n\n\tnext if (defined $name and $name ne $task);\n\n\tmy $args = \"$task <?>\";\n\tif (open CMDLINE, \"/proc/$pid/cmdline\") {\n\t\tmy $arglist = <CMDLINE>;\n\t\tif (defined $arglist) {\n\t\t\t$arglist =~ s/\\000/ /g;\n\t\t\t$args = $arglist;\n\t\t}\n\t\tclose CMDLINE;\n\t}\n\n\tmy $ppid = \"?\";\n\tif (open STAT, \"/proc/$pid/stat\") {\n\t\tmy $fields = <STAT>;\n\t\tif (defined $fields) {\n\t\t\t$ppid = (split ' ', $fields)[3];\n\t\t}\n\t\tclose STAT;\n\t}\n\n\tmy $now = strftime \"%H:%M:%S\", localtime;\n\tprintf \"%-8s %6s %6s %s\\n\", $now, $pid, $ppid, $args;\n}\n\n### end tracing\ncleanup();\n\nsub cleanup {\n\tprint \"\\nEnding tracing...\\n\";\n\tclose TPIPE;\n\twriteto \"0\", \"events/$tpdir/enable\" or\n\t    ldie \"ERROR: disabling \\\"$tptext\\\"\";\n\twriteto \"\", \"trace\";\n\tunlink $flock;\n\texit;\n}\n"
  },
  {
    "path": "deprecated/execsnoop-proc.8",
    "content": ".TH execsnoop\\-proc 8  \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nexecsnoop\\-proc \\- trace process exec() with arguments. Uses Linux ftrace. /proc version.\n.SH SYNOPSIS\n.B execsnoop\\-proc\n[\\-h] [\\-n name]\n.SH DESCRIPTION\nexecsnoop\\-proc traces process execution, showing PID, PPID, and argument details\nif possible.\n\nThis traces exec() from the fork()->exec() sequence, which means it won't\ncatch new processes that only fork(), and, it will catch processes that\nre-exec. This instruments sched:sched_process_exec without buffering, and then\nin user-space (this program) reads PPID and process arguments asynchronously\nfrom /proc.\n\nIf the process traced is very short-lived, this program may miss reading\narguments and PPID details. In that case, \"<?>\" and \"?\" will be printed\nrespectively.\n\nThis program is best-effort (a hack), and should be improved in the future when\nother kernel capabilities are made available. It may be useful in the meantime.\nIf you need a more reliable tool now, consider other tracing alternates (eg,\nSystemTap). This tool is really a proof of concept to see what ftrace can\ncurrently do.\n\nSee execsnoop(8) for another version that reads arguments from registers\ninstead of /proc.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE CONFIG and the sched:sched_process_exec tracepoint, which you may already\nhave enabled and available on recent kernels, and Perl.\n.SH OPTIONS\n\\-n name\nOnly show processes that match this name. This is filtered in user space.\n.TP\n\\-h\nPrint usage message.\n.SH EXAMPLES\n.TP\nTrace all new processes and arguments (if possible):\n.B execsnoop\\-proc\n.TP\nTrace all new processes with process name \"sed\":\n.B execsnoop\\-proc -n sed\n.SH FIELDS\n.TP\nTIME\nTime of process exec(): HH:MM:SS.\n.TP\nPID\nProcess ID.\n.TP\nPPID\nParent process ID, if this was able to be read (may be missed for short-lived\nprocesses). If it is unable to be read, \"?\" is printed.\n.TP\nARGS\nCommand line arguments, if these were able to be read in time (may be missed\nfor short-lived processes). If they are unable to be read, \"<?>\" is printed.\n.SH OVERHEAD\nThis reads and processes exec() events in user space as they occur. Since the\nrate of exec() is expected to be low (< 500/s), the overhead is expected to\nbe small or negligible.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nexecsnoop(8), top(1)\n"
  },
  {
    "path": "deprecated/execsnoop-proc_example.txt",
    "content": "Demonstrations of execsnoop-proc, the Linux ftrace version.\n\nHere's execsnoop showing what's really executed by \"man ls\":\n\n# ./execsnoop\nTIME        PID   PPID ARGS\n17:52:37  22406  25781 man ls \n17:52:37  22413  22406 preconv -e UTF-8 \n17:52:37  22416  22406 pager -s \n17:52:37  22415  22406 /bin/sh /usr/bin/nroff -mandoc -rLL=162n -rLT=162n -Tutf8 \n17:52:37  22414  22406 tbl \n17:52:37  22419  22418 locale charmap \n17:52:37  22420  22415 groff -mtty-char -Tutf8 -mandoc -rLL=162n -rLT=162n \n17:52:37  22421  22420 troff -mtty-char -mandoc -rLL=162n -rLT=162n -Tutf8 \n17:52:37  22422  22420 grotty \n\n\nThese are short-lived processes, where the argument and PPID details are often\nmissed by execsnoop:\n\n# ./execsnoop \nTIME        PID   PPID ARGS\n18:00:33  26750   1961 multilog <?>\n18:00:33  26749   1972 multilog <?>\n18:00:33  26749   1972 multilog <?>\n18:00:33  26751      ? mkdir <?>\n18:00:33  26749   1972 multilog <?>\n18:00:33  26752      ? chown <?>\n18:00:33  26750   1961 multilog <?>\n18:00:33  26750   1961 multilog <?>\n18:00:34  26753   1961 multilog <?>\n18:00:34  26754   1972 multilog <?>\n[...]\n\nThis will be fixed in a later version, but likely requires some kernel or\ntracer changes first (fetching cmdline as the probe fires).\n\n\nThe previous examples were on Linux 3.14 and 3.16 kernels. Here's a 3.2 system\nI'm running:\n\n# ./execsnoop \nERROR: enabling tracepoint \"sched:sched_process_exec\" (tracepoint missing in this kernel version?) at ./execsnoop line 78.\n\nThis kernel version is missing the sched_process_exec probe, which is pretty\nannoying.\n"
  },
  {
    "path": "disk/bitesize",
    "content": "#!/bin/bash\n#\n# bitesize - show disk I/O size as a histogram.\n#            Written using Linux perf_events (aka \"perf\").\n#\n# This can be used to characterize the distribution of block device I/O\n# sizes. To study I/O in more detail, see iosnoop(8).\n#\n# USAGE: bitesize [-h] [-b buckets] [seconds]\n#    eg,\n#        ./bitesize 10\n#\n# Run \"bitesize -h\" for full usage.\n#\n# REQUIREMENTS: perf_events and block:block_rq_issue tracepoint, which you may\n# already have on recent kernels.\n#\n# This uses multiple counting tracepoints with different filters, one for each\n# histogram bucket. While this is summarized in-kernel, the use of multiple\n# tracepoints does add addiitonal overhead, which is more evident if you add\n# more buckets. In the future this functionality will be available in an\n# efficient way in the kernel, and this tool can be rewritten.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n# \n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 22-Jul-2014\tBrendan Gregg\tCreated this.\n\nduration=0\nbuckets=(1 8 64 128)\nsecsz=512\ntrap ':' INT QUIT TERM PIPE HUP\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: bitesize [-h] [-b buckets] [seconds]\n\t                 -b buckets      # specify histogram buckets (Kbytes)\n\t                 -h              # this usage message\n\t   eg,\n\t       bitesize                  # trace I/O size until Ctrl-C\n\t       bitesize 10               # trace I/O size for 10 seconds\n\t       bitesize -b \"8 16 32\"     # specify custom bucket points\nEND\n\texit\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\n### process options\nwhile getopts b:h opt\ndo\n\tcase $opt in\n\tb)\tbuckets=($OPTARG) ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\ntpoint=block:block_rq_issue\nvar=nr_sector\nduration=$1\n\n### convert buckets (Kbytes) to disk sectors\ni=0\nsectors=(${buckets[*]})\n((max_i = ${#buckets[*]} - 1))\nwhile (( i <= max_i )); do\n\t(( sectors[$i] = ${sectors[$i]} * 1024 / $secsz ))\n\t# avoid negative array index errors for old version bash\n\tif (( i > 0 ));then\n\t\tif (( ${sectors[$i]} <= ${sectors[$i - 1]} )); then\n\t\t\tdie \"ERROR: bucket list must increase in size.\"\n\t\tfi\n\tfi\n\t(( i++ ))\ndone\n\n### build list of tracepoints and filters for each histogram bucket\nmax_b=${buckets[$max_i]}\nmax_s=${sectors[$max_i]}\ntpoints=\"-e $tpoint --filter \\\"$var < ${sectors[0]}\\\"\"\nawkarray=\ni=0\nwhile (( i < max_i )); do\n\ttpoints=\"$tpoints -e $tpoint --filter \\\"$var >= ${sectors[$i]} && \"\n\ttpoints=\"$tpoints $var < ${sectors[$i + 1]}\\\"\"\n\tawkarray=\"$awkarray buckets[$i]=${buckets[$i]};\"\n\t(( i++ ))\ndone\nawkarray=\"$awkarray buckets[$max_i]=${buckets[$max_i]};\"\ntpoints=\"$tpoints -e $tpoint --filter \\\"$var >= ${sectors[$max_i]}\\\"\"\n\n### prepare to run\nif (( duration )); then\n\tetext=\"for $duration seconds\"\n\tcmd=\"sleep $duration\"\nelse\n\tetext=\"until Ctrl-C\"\n\tcmd=\"sleep 999999\"\nfi\necho \"Tracing block I/O size (bytes), $etext...\"\n\n### run perf\nout=\"-o /dev/stdout\"\t# a workaround needed in linux 3.2; not by 3.4.15\nstat=$(eval perf stat $tpoints -a $out $cmd 2>&1)\nif (( $? != 0 )); then\n\techo >&2 \"ERROR running perf:\"\n\techo >&2 \"$stat\"\n\texit\nfi\n\n### find max value for ASCII histogram\nmost=$(echo \"$stat\" | awk -v tpoint=$tpoint '\n\t$2 == tpoint { gsub(/,/, \"\"); if ($1 > m) { m = $1 } }\n\tEND { print m }'\n)\n\n### process output\necho\necho \"$stat\" | awk -v tpoint=$tpoint -v max_i=$max_i -v most=$most '\n\tfunction star(sval, smax, swidth) {\n\t\tstars = \"\"\n\t\t# using int could avoid error on gawk\n\t\tif (int(smax) == 0) return \"\"\n\t\tfor (si = 0; si < (swidth * sval / smax); si++) {\n\t\t\tstars = stars \"#\"\n\t\t}\n\t\treturn stars\n\t}\n\tBEGIN {\n\t\t'\"$awkarray\"'\n\t\tprintf(\"            %-15s: %-8s %s\\n\", \"Kbytes\", \"I/O\",\n\t\t    \"Distribution\")\n\t}\n\t/Performance counter stats/ { i = -1 }\n\t# reverse order of rule set is important\n\t{ ok = 0 }\n\t$2 == tpoint { num = $1; gsub(/,/, \"\", num); ok = 1 }\n\tok && i >= max_i {\n\t\tprintf(\"   %10.1f -> %-10s: %-8s |%-38s|\\n\",\n\t\t    buckets[i], \"\", num, star(num, most, 38))\n\t\tnext\n\t}\n\tok && i >= 0 && i < max_i {\n\t\tprintf(\"   %10.1f -> %-10.1f: %-8s |%-38s|\\n\",\n\t\t    buckets[i], buckets[i+1] - 0.1, num,\n\t\t    star(num, most, 38))\n\t\ti++\n\t\tnext\n\t}\n\tok && i == -1 {\n\t\tprintf(\"   %10s -> %-10.1f: %-8s |%-38s|\\n\", \"\",\n\t\t    buckets[0] - 0.1, num, star(num, most, 38))\n\t\ti++\n\t}\n'\n"
  },
  {
    "path": "examples/bitesize_example.txt",
    "content": "Demonstrations of bitesize, the Linux perf_events version.\n\n\nbitesize traces block I/O issued, and reports a histogram of I/O size. By\ndefault five buckets are used to gather statistics on common I/O sizes:\n\n# ./bitesize \nTracing block I/O size (bytes), until Ctrl-C...\n^C\n            Kbytes         : I/O      Distribution\n              -> 0.9       : 0        |                                      |\n          1.0 -> 7.9       : 38       |#                                     |\n          8.0 -> 63.9      : 10108    |######################################|\n         64.0 -> 127.9     : 13       |#                                     |\n        128.0 ->           : 1        |#                                     |\n\nIn this case, most of the I/O was between 8 and 63.9 Kbytes. The \"63.9\"\nreally means \"less than 64\".\n\n\nSpecifying custom buckets to examine the I/O size in more detail:\n\n# ./bitesize -b \"8 16 24 32\"\nTracing block I/O size (bytes), until Ctrl-C...\n^C\n            Kbytes         : I/O      Distribution\n              -> 7.9       : 89       |#                                     |\n          8.0 -> 15.9      : 14665    |######################################|\n         16.0 -> 23.9      : 657      |##                                    |\n         24.0 -> 31.9      : 661      |##                                    |\n         32.0 ->           : 376      |#                                     |\n\nThe I/O is mostly between 8 and 15.9 Kbytes\n\nIt's probably 8 Kbytes. Checking:\n\n# ./bitesize -b \"8 9\"\nTracing block I/O size (bytes), until Ctrl-C...\n^C\n            Kbytes         : I/O      Distribution\n              -> 7.9       : 62       |#                                     |\n          8.0 -> 8.9       : 11719    |######################################|\n          9.0 ->           : 1358     |#####                                 |\n\nIt is.\n\nThe overhead of this tool is relative to the number of buckets used, hence only\nusing what is necessary.\n\nTo study this I/O in more detail, I can use iosnoop(8) and capture it to a file\nfor post-processing.\n\n\nUse -h to print the USAGE message:\n\n# ./bitesize -h\nUSAGE: bitesize [-h] [-b buckets] [seconds]\n                 -b buckets      # specify histogram buckets (Kbytes)\n                 -h              # this usage message\n   eg,\n       bitesize                  # trace I/O size until Ctrl-C\n       bitesize 10               # trace I/O size for 10 seconds\n       bitesize -b \"8 16 32\"     # specify custom bucket points\n"
  },
  {
    "path": "examples/cachestat_example.txt",
    "content": "Demonstrations of cachestat, the Linux ftrace version.\n\n\nHere is some sample output showing file system cache statistics, followed by\nthe workload that caused it:\n\n# ./cachestat -t\nCounting cache functions... Output every 1 seconds.\nTIME         HITS   MISSES  DIRTIES    RATIO   BUFFERS_MB   CACHE_MB\n08:28:57      415        0        0   100.0%            1        191\n08:28:58      411        0        0   100.0%            1        191\n08:28:59      362       97        0    78.9%            0          8\n08:29:00      411        0        0   100.0%            0          9\n08:29:01      775    20489        0     3.6%            0         89\n08:29:02      411        0        0   100.0%            0         89\n08:29:03     6069        0        0   100.0%            0         89\n08:29:04    15249        0        0   100.0%            0         89\n08:29:05      411        0        0   100.0%            0         89\n08:29:06      411        0        0   100.0%            0         89\n08:29:07      411        0        3   100.0%            0         89\n[...]\n\nI used the -t option to include the TIME column, to make describing the output\neasier.\n\nThe workload was:\n\n# echo 1 > /proc/sys/vm/drop_caches; sleep 2; cksum 80m; sleep 2; cksum 80m\n\nAt 8:28:58, the page cache was dropped by the first command, which can be seen\nby the drop in size for \"CACHE_MB\" (page cache size) from 191 Mbytes to 8.\nAfter a 2 second sleep, a cksum command was issued at 8:29:01, for an 80 Mbyte\nfile (called \"80m\"), which caused a total of ~20,400 misses (\"MISSES\" column),\nand the page cache size to grow by 80 Mbytes. The hit ratio during this dropped\nto 3.6%. Finally, after another 2 second sleep, at 8:29:03 the cksum command\nwas run a second time, this time hitting entirely from cache.\n\nInstrumenting all file system cache accesses does cost some overhead, and this\ntool might slow your target system by 2% or so. Test before use if this is a\nconcern.\n\nThis tool also uses dynamic tracing, and is tied to Linux kernel implementation\ndetails. If it doesn't work for you, it probably needs fixing.\n\n\nUse -h to print the USAGE message:\n\n# ./cachestat -h\nUSAGE: cachestat [-Dht] [interval]\n                 -D              # print debug counters\n                 -h              # this usage message\n                 -t              # include timestamp\n                 interval        # output interval in secs (default 1)\n  eg,\n       cachestat                 # show stats every second\n       cachestat 5               # show stats every 5 seconds\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/execsnoop_example.txt",
    "content": "Demonstrations of execsnoop, the Linux ftrace version.\n\n\nHere's execsnoop showing what's really executed by \"man ls\":\n\n# ./execsnoop \nTracing exec()s. Ctrl-C to end.\n   PID   PPID ARGS\n 22898  22004 man ls\n 22905  22898 preconv -e UTF-8\n 22908  22898 pager -s\n 22907  22898 nroff -mandoc -rLL=164n -rLT=164n -Tutf8\n 22906  22898 tbl\n 22911  22910 locale charmap\n 22912  22907 groff -mtty-char -Tutf8 -mandoc -rLL=164n -rLT=164n\n 22913  22912 troff -mtty-char -mandoc -rLL=164n -rLT=164n -Tutf8\n 22914  22912 grotty\n\nMany commands. This is particularly useful for understanding application\nstartup.\n\n\nAnother use for execsnoop is identifying short-lived processes. Eg, with the -t\noption to see timestamps:\n\n# ./execsnoop -t\nTracing exec()s. Ctrl-C to end.\nTIMEs               PID   PPID ARGS\n7419756.154031     8185   8181 mawk -W interactive -v o=1 -v opt_name=0 -v name= [...]\n7419756.154131     8186   8184 cat -v trace_pipe\n7419756.245264     8188   1698 ./run\n7419756.245691     8189   1696 ./run\n7419756.246212     8187   1689 ./run\n7419756.278993     8190   1693 ./run\n7419756.278996     8191   1692 ./run\n7419756.288430     8192   1695 ./run\n7419756.290115     8193   1691 ./run\n7419756.292406     8194   1699 ./run\n7419756.293986     8195   1690 ./run\n7419756.294149     8196   1686 ./run\n7419756.296527     8197   1687 ./run\n7419756.296973     8198   1697 ./run\n7419756.298356     8200   1685 ./run\n7419756.298683     8199   1688 ./run\n7419757.269883     8201   1696 ./run\n[...]\n\nSo we're running many \"run\" commands every second. The PPID is included, so I\ncan debug this further (they are \"supervise\" processes).\n\nShort-lived processes can consume CPU and not be visible from top(1), and can\nbe the source of hidden performance issues.\n\n\nHere's another example: I noticed CPU usage was high in top(1), but couldn't\nsee the responsible process:\n\n$ top\ntop - 00:04:32 up 78 days, 15:41,  3 users,  load average: 0.85, 0.29, 0.14\nTasks: 123 total,   1 running, 121 sleeping,   0 stopped,   1 zombie\nCpu(s): 15.7%us, 34.9%sy,  0.0%ni, 49.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.2%st\nMem:   7629464k total,  7537216k used,    92248k free,  1376492k buffers\nSwap:        0k total,        0k used,        0k free,  5432356k cached\n\n  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND\n 7225 bgregg-t  20   0 29480 6196 2128 S    3  0.1   0:02.64 ec2rotatelogs\n    1 root      20   0 24320 2256 1340 S    0  0.0   0:01.23 init\n    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd\n    3 root      20   0     0    0    0 S    0  0.0   1:19.61 ksoftirqd/0\n    4 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/0:0\n    5 root      20   0     0    0    0 S    0  0.0   0:00.01 kworker/u:0\n    6 root      RT   0     0    0    0 S    0  0.0   0:16.00 migration/0\n    7 root      RT   0     0    0    0 S    0  0.0   0:17.29 watchdog/0\n    8 root      RT   0     0    0    0 S    0  0.0   0:15.85 migration/1\n    9 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/1:0\n[...]\n\nSee the line starting with \"Cpu(s):\". So there's about 50% CPU utilized (this\nis a two CPU server, so that's equivalent to one full CPU), but this CPU usage\nisn't visible from the process listing.\n\nvmstat agreed, showing the same average CPU usage statistics:\n \n# vmstat 1\nprocs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----\n r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa\n 2  0      0  92816 1376476 5432188    0    0     0     3    2    1  0  1 99  0\n 1  0      0  92676 1376484 5432264    0    0     0    24 6573 6130 12 38 49  0\n 1  0      0  91964 1376484 5432272    0    0     0     0 6529 6097 16 35 49  0\n 1  0      0  92692 1376484 5432272    0    0     0     0 6192 5775 17 35 49  0\n 1  0      0  92692 1376484 5432272    0    0     0     0 6554 6121 14 36 50  0\n 1  0      0  91940 1376484 5432272    0    0     0    12 6546 6101 13 38 49  0\n 1  0      0  92560 1376484 5432272    0    0     0     0 6201 5769 15 35 49  0\n 1  0      0  92676 1376484 5432272    0    0     0     0 6524 6123 17 34 49  0\n 1  0      0  91932 1376484 5432272    0    0     0     0 6546 6107 10 40 49  0\n 1  0      0  92832 1376484 5432272    0    0     0     0 6057 5710 13 38 49  0\n 1  0      0  92248 1376484 5432272    0    0    84    28 6592 6183 16 36 48  1\n 1  0      0  91504 1376492 5432348    0    0     0    12 6540 6098 18 33 49  1\n[...]\n\nSo this could be caused by short-lived processes, who vanish before they are\nseen by top(1). Do I have my execsnoop handy? Yes:\n\n# ~/perf-tools/bin/execsnoop \nTracing exec()s. Ctrl-C to end.\n   PID   PPID ARGS\n 10239  10229 gawk -v o=0 -v opt_name=0 -v name= -v opt_duration=0 [...]\n 10240  10238 cat -v trace_pipe\n 10242   7225 sh [?]\n 10243  10242 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.201201.3122.txt\n 10245   7225 sh [?]\n 10246  10245 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.202201.3122.txt\n 10248   7225 sh [?]\n 10249  10248 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.203201.3122.txt\n 10251   7225 sh [?]\n 10252  10251 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.204201.3122.txt\n 10254   7225 sh [?]\n 10255  10254 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.205201.3122.txt\n 10257   7225 sh [?]\n 10258  10257 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.210201.3122.txt\n 10260   7225 sh [?]\n 10261  10260 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.211201.3122.txt\n 10263   7225 sh [?]\n 10264  10263 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.212201.3122.txt\n 10266   7225 sh [?]\n 10267  10266 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.213201.3122.txt\n[...]\n\nThe output scrolled quickly, showing that many shell and lsof processes were\nbeing launched. If you check the PID and PPID columns carefully, you can see that\nthese are ultimately all from PID 7225. We saw that earlier in the top output:\nec2rotatelogs, at 3% CPU. I now know the culprit.\n\nI should have used \"-t\" to show the timestamps with this example.\n\n\nRun -h to print the USAGE message:\n\n# ./execsnoop -h\nUSAGE: execsnoop [-hrt] [-a argc] [-d secs] [name]\n                 -d seconds      # trace duration, and use buffers\n                 -a argc         # max args to show (default 8)\n                 -r              # include re-execs\n                 -t              # include time (seconds)\n                 -h              # this usage message\n                 name            # process name to match (REs allowed)\n  eg,\n       execsnoop                 # watch exec()s live (unbuffered)\n       execsnoop -d 1            # trace 1 sec (buffered)\n       execsnoop grep            # trace process names containing grep\n       execsnoop 'log$'          # filenames ending in \"log\"\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/funccount_example.txt",
    "content": "Demonstrations of funccount, the Linux ftrace version.\n\n\nTracing all kernel functions that start with \"bio_\" (which would be block\ninterface functions), and counting how many times they were executed until\nCtrl-C is hit:\n\n# ./funccount 'bio_*'\nTracing \"bio_*\"... Ctrl-C to end.\n^C\nFUNC                              COUNT\nbio_attempt_back_merge               26\nbio_get_nr_vecs                     361\nbio_alloc                           536\nbio_alloc_bioset                    536\nbio_endio                           536\nbio_free                            536\nbio_fs_destructor                   536\nbio_init                            536\nbio_integrity_enabled               536\nbio_put                             729\nbio_add_page                       1004\n\nNote that these counts are performed in-kernel context, using the ftrace\nfunction profiler, which means this is a (relatively) low overhead technique.\nTest yourself to quantify overhead.\n\n\nAs was demonstrated here, wildcards can be used. Individual functions can also\nbe specified. For example, all of the following are valid arguments:\n\n\tbio_init\n\tbio_*\n\t*init\n\t*bio*\n\nA \"*\" within a string (eg, \"bio*init\") is not supported.\n\nThe full list of what can be traced is in:\n/sys/kernel/debug/tracing/available_filter_functions, which can be grep'd to\ncheck what is there. Note that grep uses regular expressions, whereas\nfunccount uses globbing for wildcards.\n\n\nCounting all \"tcp_\" kernel functions, and printing a summary every one second:\n\n# ./funccount -i 1 -t 5 'tcp_*'\nTracing \"tcp_*\". Top 5 only... Ctrl-C to end.\n\nFUNC                              COUNT\ntcp_cleanup_rbuf                    386\ntcp_service_net_dma                 386\ntcp_established_options             549\ntcp_v4_md5_lookup                   560\ntcp_v4_md5_do_lookup                890\n\nFUNC                              COUNT\ntcp_service_net_dma                 498\ntcp_cleanup_rbuf                    499\ntcp_established_options             664\ntcp_v4_md5_lookup                   672\ntcp_v4_md5_do_lookup               1071\n\n[...]\n\nNeat.\n\n\nTracing all \"ext4*\" kernel functions for 10 seconds, and printing the top 25:\n\n# ./funccount -t 25 -d 10 'ext4*'\nTracing \"ext4*\" for 10 seconds. Top 25 only...\n\nFUNC                              COUNT\next4_inode_bitmap                   840\next4_meta_trans_blocks              840\next4_ext_drop_refs                  843\next4_find_entry                     845\next4_discard_preallocations        1008\next4_free_inodes_count             1120\next4_group_desc_csum               1120\next4_group_desc_csum_set           1120\next4_getblk                        1128\next4_es_free_extent                1328\next4_map_blocks                    1471\next4_es_lookup_extent              1751\next4_mb_check_limits               1873\next4_es_lru_add                    2031\next4_data_block_valid              2312\next4_journal_check_start           3080\next4_mark_inode_dirty              5320\next4_get_inode_flags               5955\next4_get_inode_loc                 5955\next4_mark_iloc_dirty               5955\next4_reserve_inode_write           5955\next4_inode_table                   7076\next4_get_group_desc                8476\next4_has_inline_data               9492\next4_inode_touch_time_cmp         38980\n\nEnding tracing...\n\nSo ext4_inode_touch_time_cmp() was called the most frequently, at 38,980 times.\nThis may be normal, this may not. The purpose of this tool is to give you one\nview of how one or many kernel functions are executed. Previously I had little\nidea what ext4 was doing internally. Now I know the top 25 functions, and their\nrate, and can begin researching them from the source code.\n\n\nUse -h to print the USAGE message:\n\n# ./funccount -h\nUSAGE: funccount [-hT] [-i secs] [-d secs] [-t top] funcstring\n                 -d seconds      # total duration of trace\n                 -h              # this usage message\n                 -i seconds      # interval summary\n                 -t top          # show top num entries only\n                 -T              # include timestamp (for -i)\n  eg,\n       funccount 'vfs*'          # trace all funcs that match \"vfs*\"\n       funccount -d 5 'tcp*'     # trace \"tcp*\" funcs for 5 seconds\n       funccount -t 10 'ext3*'   # show top 10 \"ext3*\" funcs\n       funccount -i 1 'ext3*'    # summary every 1 second\n       funccount -i 1 -d 5 'ext3*' # 5 x 1 second summaries\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/funcgraph_example.txt",
    "content": "Demonstrations of funcgraph, the Linux ftrace version.\n\n\nI'll start by showing do_nanosleep(), since it's usually a low frequency\nfunction that can be easily triggered (run \"vmstat 1\"):\n\n# ./funcgraph do_nanosleep\nTracing \"do_nanosleep\"... Ctrl-C to end.\n 0)               |  do_nanosleep() {\n 0)               |    hrtimer_start_range_ns() {\n 0)               |      __hrtimer_start_range_ns() {\n 0)               |        lock_hrtimer_base.isra.24() {\n 0)   0.198 us    |          _raw_spin_lock_irqsave();\n 0)   0.908 us    |        }\n 0)   0.061 us    |        idle_cpu();\n 0)   0.117 us    |        ktime_get();\n 0)   0.371 us    |        enqueue_hrtimer();\n 0)   0.075 us    |        _raw_spin_unlock_irqrestore();\n 0)   3.447 us    |      }\n 0)   3.998 us    |    }\n 0)               |    schedule() {\n 0)               |      __schedule() {\n 0)   0.050 us    |        rcu_note_context_switch();\n 0)   0.055 us    |        _raw_spin_lock_irq();\n 0)               |        deactivate_task() {\n 0)               |          dequeue_task() {\n 0)   0.142 us    |            update_rq_clock();\n 0)               |            dequeue_task_fair() {\n 0)               |              dequeue_entity() {\n 0)               |                update_curr() {\n 0)   0.086 us    |                  cpuacct_charge();\n 0)   0.757 us    |                }\n 0)   0.052 us    |                clear_buddies();\n 0)   0.103 us    |                update_cfs_load();\n 0)               |                update_cfs_shares() {\n 0)               |                  reweight_entity() {\n 0)   0.077 us    |                    update_curr();\n 0)   0.438 us    |                  }\n 0)   0.794 us    |                }\n 0)   3.067 us    |              }\n 0)   0.064 us    |              set_next_buddy();\n 0)   0.066 us    |              update_cfs_load();\n 0)   0.085 us    |              update_cfs_shares();\n 0)               |              hrtick_update() {\n 0)   0.063 us    |                hrtick_start_fair();\n 0)   0.367 us    |              }\n 0)   5.188 us    |            }\n 0)   5.923 us    |          }\n 0)   6.228 us    |        }\n 0)               |        put_prev_task_fair() {\n 0)   0.078 us    |          put_prev_entity();\n 0)               |          put_prev_entity() {\n 0)   0.070 us    |            update_curr();\n 0)   0.074 us    |            __enqueue_entity();\n 0)   0.737 us    |          }\n 0)   1.367 us    |        }\n 0)               |        pick_next_task_fair() {\n 0)               |          pick_next_entity() {\n 0)   0.052 us    |            wakeup_preempt_entity.isra.95();\n 0)   0.070 us    |            clear_buddies();\n 0)   0.676 us    |          }\n 0)               |          set_next_entity() {\n 0)   0.052 us    |            update_stats_wait_end();\n 0)   0.435 us    |          }\n 0)               |          pick_next_entity() {\n 0)   0.065 us    |            clear_buddies();\n 0)   0.376 us    |          }\n 0)               |          set_next_entity() {\n 0)   0.067 us    |            update_stats_wait_end();\n 0)   0.374 us    |          }\n 0)   0.051 us    |          hrtick_start_fair();\n 0)   3.879 us    |        }\n 0)   0.057 us    |        paravirt_start_context_switch();\n 0)               |        xen_load_sp0() {\n 0)   0.050 us    |          paravirt_get_lazy_mode();\n 0)   0.057 us    |          __xen_mc_entry();\n 0)   0.056 us    |          paravirt_get_lazy_mode();\n 0)   1.441 us    |        }\n 0)               |        xen_load_tls() {\n 0)   0.049 us    |          paravirt_get_lazy_mode();\n 0)   0.051 us    |          paravirt_get_lazy_mode();\n 0)               |          load_TLS_descriptor() {\n 0)               |            arbitrary_virt_to_machine() {\n 0)   0.081 us    |              __virt_addr_valid();\n 0)   0.052 us    |              __phys_addr();\n 0)   0.084 us    |              get_phys_to_machine();\n 0)   1.115 us    |            }\n 0)   0.053 us    |            __xen_mc_entry();\n 0)   1.744 us    |          }\n 0)               |          load_TLS_descriptor() {\n 0)               |            arbitrary_virt_to_machine() {\n 0)   0.053 us    |              __virt_addr_valid();\n 0)   0.056 us    |              __phys_addr();\n 0)   0.057 us    |              get_phys_to_machine();\n 0)   0.990 us    |            }\n 0)   0.053 us    |            __xen_mc_entry();\n 0)   1.583 us    |          } /* load_TLS_descriptor */\n 0)               |          load_TLS_descriptor() {\n 0)               |            arbitrary_virt_to_machine() {\n 0)   0.057 us    |              __virt_addr_valid();\n 0)   0.051 us    |              __phys_addr();\n 0)   0.053 us    |              get_phys_to_machine();\n 0)   0.978 us    |            }\n 0)   0.052 us    |            __xen_mc_entry();\n 0)   1.586 us    |          }\n 0)   0.052 us    |          paravirt_get_lazy_mode();\n 0)   6.630 us    |        }\n 0)               |        xen_end_context_switch() {\n 0)   0.666 us    |          xen_mc_flush();\n 0)   0.050 us    |          paravirt_end_context_switch();\n 0)   1.286 us    |        }\n 0)   0.172 us    |        xen_write_msr_safe();\n ------------------------------------------\n 0)  platfor-3210  =>  vmstat-2854  \n ------------------------------------------\n\n 0)               |  do_nanosleep() {\n 0)               |    hrtimer_start_range_ns() {\n 0)               |      __hrtimer_start_range_ns() {\n 0)               |        lock_hrtimer_base.isra.24() {\n 0)   0.217 us    |          _raw_spin_lock_irqsave();\n 0)   0.831 us    |        }\n 0)   0.066 us    |        idle_cpu();\n 0)   0.123 us    |        ktime_get();\n 0)   1.172 us    |        enqueue_hrtimer();\n 0)   0.089 us    |        _raw_spin_unlock_irqrestore();\n 0)   4.050 us    |      }\n 0)   4.523 us    |    }\n[...]\n\nThe default output shows the function call graph, including all child kernel\nfunctions, along with the function duration times. These times are printed on\neither the return line for the function (\"}\"), or for leaf functions, on the\nsame line.\n\nThe format of this output is documented in the function graph section of the\nkernel source file Documentation/trace/ftrace.txt.\n\nThis particular example shows the workings of do_nanosleep, in the first dozen\nlines, and then schedule() is called to sleep this thread and run another. The\ninner workings of schedule() is included in the output.\n\nThis output is great for determining the behavior of a certain kernel function,\nand to identify functions that can be studied in more details using other, lower\noverhead, tools (eg, funccount(8), functrace(8), kprobe(8)). The overheads\nof funcgraph are moderate, since all kernel functions are traced in case\nthey are executed, then included in the output if they are.\n\nNow, if you want to start understanding the general behavior of the kernel,\nwithout a certain kernel function in mind, you may be better to begin with\nCPU stack profiling using perf and generating a flame graph. Such an approach\nhas low overhead, as you are in control of the frequency of event collection\n(eg, gathering CPU stacks at 99 Hertz). For instructions, see:\nhttp://www.brendangregg.com/perf.html#FlameGraphs\n\n\n\nOn newer Linux kernels, you can use the -m option to limit the function\ndepth. Eg, 3 levels only:\n\n# ./funcgraph -m 3 do_nanosleep\nTracing \"do_nanosleep\"... Ctrl-C to end.\n 1)               |  do_nanosleep() {\n 1)               |    hrtimer_start_range_ns() {\n 1)   1.115 us    |      __hrtimer_start_range_ns();\n 1)   1.919 us    |    }\n 1)               |    schedule() {\n 1)               |      __schedule() {\n 1)   1000131 us |      }\n 1)   11.006 us   |      xen_evtchn_do_upcall();\n 1)   1000149 us |    }\n 1)               |    hrtimer_cancel() {\n 1)   0.212 us    |      hrtimer_try_to_cancel();\n 1)   0.699 us    |    }\n 1)   1000154 us |  }\n\nNeat.\n\n\nNow do_sys_open() to 3 levels:\n\n 0)               |  do_sys_open() {\n 0)               |    getname() {\n 0)   0.296 us    |      getname_flags();\n 0)   0.768 us    |    }\n 0)               |    get_unused_fd_flags() {\n 0)   0.397 us    |      __alloc_fd();\n 0)   0.827 us    |    }\n 0)               |    do_filp_open() {\n 0)   4.166 us    |      path_openat();\n 0)   4.617 us    |    }\n 0)               |    __fsnotify_parent() {\n 0)   0.083 us    |      dget_parent();\n 0)   0.063 us    |      dput();\n 0)   0.883 us    |    }\n 0)   0.058 us    |    fsnotify();\n 0)               |    fd_install() {\n 0)   0.133 us    |      __fd_install();\n 0)   0.525 us    |    }\n 0)               |    putname() {\n 0)   0.198 us    |      final_putname();\n 0)   0.512 us    |    }\n 0)   10.777 us   |  }\n[...]\n\nI can then pick the highest latency child function, then run funcgraph again\nusing it as the target.\n\n\nWithout timestamps (-D to elide duration):\n\n# ./funcgraph -Dm 3 do_sys_open\nTracing \"do_sys_open\"... Ctrl-C to end.\n 1) do_sys_open() {\n 1)   getname() {\n 1)     getname_flags();\n 1)   }\n 1)   get_unused_fd_flags() {\n 1)     __alloc_fd();\n 1)   }\n 1)   do_filp_open() {\n 1)     path_openat();\n 1)   }\n 1)   __fsnotify_parent();\n 1)   fsnotify();\n 1)   fd_install() {\n 1)     __fd_install();\n 1)   }\n 1)   putname() {\n 1)     final_putname();\n 1)   }\n 1) }\n\nBeautiful.\n\nI could elide the CPU column as well, but I want to leave it: if it changes\nhalf-way through some output, you know the CPU buffer has switched, and the\noutput may be shuffled.\n\n\nFor this example, I trace vfs_read() calls by process ID 5363: which is a bash\nshell. I also include headers (-H) and absolute timestamps (-t). While\ntracing, in that bash shell, I typed the word \"hello\":\n\n# ./funcgraph -Htp 5363 vfs_read\nTracing \"vfs_read\" for PID 5363... Ctrl-C to end.\n# tracer: function_graph\n#\n#     TIME        CPU  DURATION                  FUNCTION CALLS\n#      |          |     |   |                     |   |   |   |\n7238523.638008 |   0)               |              finish_task_switch() {\n7238523.638012 |   0)               |                xen_evtchn_do_upcall() {\n7238523.638012 |   0)               |                  irq_enter() {\n7238523.638013 |   0)   0.153 us    |                    rcu_irq_enter();\n7238523.638014 |   0)   1.144 us    |                  }\n7238523.638014 |   0)   0.056 us    |                  exit_idle();\n7238523.638014 |   0)               |                  __xen_evtchn_do_upcall() {\n7238523.638015 |   0)               |                    evtchn_2l_handle_events() {\n7238523.638015 |   0)   0.057 us    |                      irq_from_virq();\n7238523.638015 |   0)               |                      evtchn_from_irq() {\n7238523.638015 |   0)               |                        irq_get_irq_data() {\n7238523.638016 |   0)   0.058 us    |                          irq_to_desc();\n7238523.638016 |   0)   0.565 us    |                        }\n7238523.638016 |   0)   0.966 us    |                      }\n7238523.638016 |   0)               |                      get_evtchn_to_irq() {\n7238523.638017 |   0)   0.050 us    |                        evtchn_2l_max_channels();\n7238523.638017 |   0)   0.386 us    |                      }\n7238523.638017 |   0)               |                      generic_handle_irq() {\n7238523.638017 |   0)   0.058 us    |                        irq_to_desc();\n7238523.638018 |   0)               |                        handle_percpu_irq() {\n7238523.638018 |   0)               |                          ack_dynirq() {\n7238523.638018 |   0)               |                            evtchn_from_irq() {\n7238523.638018 |   0)               |                              irq_get_irq_data() {\n7238523.638019 |   0)   0.049 us    |                                irq_to_desc();\n7238523.638019 |   0)   0.441 us    |                              }\n7238523.638019 |   0)   0.772 us    |                            }\n7238523.638019 |   0)   0.049 us    |                            irq_move_irq();\n7238523.638020 |   0)   0.060 us    |                            evtchn_2l_clear_pending();\n7238523.638020 |   0)   1.810 us    |                          }\n7238523.638020 |   0)               |                          handle_irq_event_percpu() {\n7238523.638020 |   0)               |                            xen_irq_work_interrupt() {\n7238523.638021 |   0)               |                              irq_enter() {\n7238523.638021 |   0)   0.056 us    |                                rcu_irq_enter();\n7238523.638021 |   0)   0.384 us    |                              }\n7238523.638021 |   0)               |                              __wake_up() {\n7238523.638022 |   0)   0.059 us    |                                _raw_spin_lock_irqsave();\n7238523.638022 |   0)               |                                __wake_up_common() {\n7238523.638022 |   0)               |                                  autoremove_wake_function() {\n7238523.638023 |   0)               |                                    default_wake_function() {\n7238523.638023 |   0)               |                                      try_to_wake_up() {\n7238523.638023 |   0)   0.220 us    |                                        _raw_spin_lock_irqsave();\n7238523.638024 |   0)   0.270 us    |                                        task_waking_fair();\n7238523.638024 |   0)               |                                        select_task_rq_fair() {\n7238523.638025 |   0)   0.055 us    |                                          source_load();\n7238523.638025 |   0)   0.056 us    |                                          target_load();\n7238523.638025 |   0)   0.060 us    |                                          idle_cpu();\n7238523.638026 |   0)   0.054 us    |                                          cpus_share_cache();\n7238523.638026 |   0)   0.083 us    |                                          idle_cpu();\n7238523.638026 |   0)   2.060 us    |                                        }\n7238523.638027 |   0)   0.051 us    |                                        _raw_spin_lock();\n7238523.638027 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7238523.638027 |   0)               |                                          activate_task() {\n7238523.638027 |   0)               |                                            enqueue_task() {\n7238523.638028 |   0)   0.120 us    |                                              update_rq_clock();\n7238523.638028 |   0)               |                                              enqueue_task_fair() {\n7238523.638028 |   0)               |                                                enqueue_entity() {\n7238523.638028 |   0)   0.147 us    |                                                  update_curr();\n7238523.638029 |   0)   0.055 us    |                                                  __compute_runnable_contrib.part.51();\n7238523.638029 |   0)   0.066 us    |                                                  __update_entity_load_avg_contrib();\n7238523.638029 |   0)   0.141 us    |                                                  update_cfs_rq_blocked_load();\n7238523.638030 |   0)   0.068 us    |                                                  account_entity_enqueue();\n7238523.638030 |   0)   0.351 us    |                                                  update_cfs_shares();\n7238523.638031 |   0)   0.053 us    |                                                  place_entity();\n7238523.638031 |   0)   0.082 us    |                                                  __enqueue_entity();\n7238523.638032 |   0)   0.050 us    |                                                  update_cfs_rq_blocked_load();\n7238523.638032 |   0)   3.922 us    |                                                }\n7238523.638032 |   0)               |                                                enqueue_entity() {\n7238523.638033 |   0)   0.058 us    |                                                  update_curr();\n7238523.638033 |   0)   0.056 us    |                                                  __compute_runnable_contrib.part.51();\n7238523.638033 |   0)   0.078 us    |                                                  __update_entity_load_avg_contrib();\n7238523.638034 |   0)   0.055 us    |                                                  update_cfs_rq_blocked_load();\n7238523.638034 |   0)   0.064 us    |                                                  account_entity_enqueue();\n7238523.638034 |   0)   0.059 us    |                                                  update_cfs_shares();\n7238523.638035 |   0)   0.050 us    |                                                  place_entity();\n7238523.638036 |   0)   0.057 us    |                                                  __enqueue_entity();\n7238523.638036 |   0)   3.829 us    |                                                }\n7238523.638037 |   0)   0.057 us    |                                                hrtick_update();\n7238523.638037 |   0)   8.876 us    |                                              }\n7238523.638037 |   0)   9.698 us    |                                            }\n7238523.638037 |   0)   10.113 us   |                                          }\n7238523.638038 |   0)               |                                          ttwu_do_wakeup() {\n7238523.638038 |   0)               |                                            check_preempt_curr() {\n7238523.638038 |   0)               |                                              resched_task() {\n7238523.638038 |   0)               |                                                xen_smp_send_reschedule() {\n7238523.638038 |   0)               |                                                  xen_send_IPI_one() {\n7238523.638039 |   0)               |                                                    notify_remote_via_irq() {\n7238523.638039 |   0)               |                                                      evtchn_from_irq() {\n7238523.638039 |   0)               |                                                        irq_get_irq_data() {\n7238523.638039 |   0)   0.051 us    |                                                          irq_to_desc();\n7238523.638039 |   0)   0.518 us    |                                                        }\n7238523.638040 |   0)   0.955 us    |                                                      }\n7238523.638041 |   0)   2.001 us    |                                                    }\n7238523.638041 |   0)   2.391 us    |                                                  }\n7238523.638041 |   0)   2.745 us    |                                                }\n7238523.638041 |   0)   3.183 us    |                                              }\n7238523.638042 |   0)   3.663 us    |                                            }\n7238523.638042 |   0)   4.621 us    |                                          }\n7238523.638043 |   0)   15.443 us   |                                        }\n7238523.638043 |   0)   0.067 us    |                                        _raw_spin_unlock();\n7238523.638043 |   0)   0.167 us    |                                        ttwu_stat();\n7238523.638044 |   0)   0.087 us    |                                        _raw_spin_unlock_irqrestore();\n7238523.638044 |   0)   21.447 us   |                                      }\n7238523.638045 |   0)   21.940 us   |                                    }\n7238523.638045 |   0)   22.406 us   |                                  }\n7238523.638045 |   0)   23.071 us   |                                }\n7238523.638045 |   0)   0.073 us    |                                _raw_spin_unlock_irqrestore();\n7238523.638046 |   0)   24.382 us   |                              }\n7238523.638046 |   0)               |                              irq_exit() {\n7238523.638047 |   0)   0.085 us    |                                idle_cpu();\n7238523.638047 |   0)   0.093 us    |                                rcu_irq_exit();\n7238523.638048 |   0)   1.242 us    |                              }\n7238523.638048 |   0)   27.410 us   |                            }\n7238523.638049 |   0)   0.139 us    |                            add_interrupt_randomness();\n7238523.638049 |   0)   0.089 us    |                            note_interrupt();\n7238523.638050 |   0)   29.582 us   |                          }\n7238523.638050 |   0)   32.112 us   |                        }\n7238523.638050 |   0)   32.951 us   |                      }\n7238523.638051 |   0)   35.765 us   |                    }\n7238523.638051 |   0)   36.170 us   |                  }\n7238523.638051 |   0)               |                  irq_exit() {\n7238523.638051 |   0)   0.082 us    |                    idle_cpu();\n7238523.638052 |   0)   0.071 us    |                    rcu_irq_exit();\n7238523.638053 |   0)   1.328 us    |                  }\n7238523.638053 |   0)   40.563 us   |                }\n7238523.638054 |   0)               |                __mmdrop() {\n7238523.638054 |   0)               |                  pgd_free() {\n7238523.638055 |   0)   0.151 us    |                    _raw_spin_lock();\n7238523.638055 |   0)   0.069 us    |                    _raw_spin_unlock();\n7238523.638056 |   0)               |                    xen_pgd_free() {\n7238523.638056 |   0)   0.067 us    |                      xen_get_user_pgd();\n7238523.638057 |   0)               |                      free_pages() {\n7238523.638057 |   0)               |                        __free_pages() {\n7238523.638057 |   0)               |                          free_hot_cold_page() {\n7238523.638058 |   0)   0.080 us    |                            free_pages_prepare();\n7238523.638058 |   0)   0.363 us    |                            get_pfnblock_flags_mask();\n7238523.638059 |   0)   1.626 us    |                          }\n7238523.638059 |   0)   2.317 us    |                        }\n7238523.638060 |   0)   2.847 us    |                      }\n7238523.638060 |   0)   3.908 us    |                    }\n7238523.638060 |   0)               |                    free_pages() {\n7238523.638060 |   0)               |                      __free_pages() {\n7238523.638061 |   0)               |                        free_hot_cold_page() {\n7238523.638061 |   0)   0.083 us    |                          free_pages_prepare();\n7238523.638061 |   0)   0.139 us    |                          get_pfnblock_flags_mask();\n7238523.638062 |   0)   1.062 us    |                        }\n7238523.638062 |   0)   1.534 us    |                      }\n7238523.638062 |   0)   2.038 us    |                    }\n7238523.638063 |   0)   8.268 us    |                  }\n7238523.638064 |   0)   0.160 us    |                  destroy_context();\n7238523.638065 |   0)   0.384 us    |                  kmem_cache_free();\n7238523.638066 |   0)   11.433 us   |                }\n7238523.638066 |   0)   54.448 us   |              }\n7238523.638066 |   0)   19354026 us |            } /* __schedule */\n7238523.638067 |   0)   19354026 us |          } /* schedule */\n7238523.638067 |   0)   19354027 us |        } /* schedule_timeout */\n7238523.638067 |   0)   0.121 us    |        down_read();\n7238523.638068 |   0)               |        copy_from_read_buf() {\n7238523.638069 |   0)               |          tty_audit_add_data() {\n7238523.638070 |   0)   0.220 us    |            _raw_spin_lock_irqsave();\n7238523.638071 |   0)   0.097 us    |            _raw_spin_unlock_irqrestore();\n7238523.638071 |   0)   0.078 us    |            _raw_spin_lock_irqsave();\n7238523.638072 |   0)   0.077 us    |            _raw_spin_unlock_irqrestore();\n7238523.638072 |   0)   2.795 us    |          }\n7238523.638073 |   0)   4.183 us    |        }\n7238523.638073 |   0)   0.084 us    |        copy_from_read_buf();\n7238523.638074 |   0)   0.078 us    |        n_tty_set_room();\n7238523.638074 |   0)   0.082 us    |        n_tty_write_wakeup();\n7238523.638075 |   0)               |        __wake_up() {\n7238523.638075 |   0)   0.084 us    |          _raw_spin_lock_irqsave();\n7238523.638076 |   0)               |          __wake_up_common() {\n7238523.638076 |   0)   0.095 us    |            pollwake();\n7238523.638077 |   0)   0.819 us    |          }\n7238523.638077 |   0)   0.074 us    |          _raw_spin_unlock_irqrestore();\n7238523.638078 |   0)   2.463 us    |        }\n7238523.638078 |   0)   0.071 us    |        n_tty_set_room();\n7238523.638078 |   0)   0.082 us    |        up_read();\n7238523.638079 |   0)               |        remove_wait_queue() {\n7238523.638079 |   0)   0.082 us    |          _raw_spin_lock_irqsave();\n7238523.638080 |   0)   0.086 us    |          _raw_spin_unlock_irqrestore();\n7238523.638080 |   0)   1.239 us    |        }\n7238523.638081 |   0)   0.142 us    |        mutex_unlock();\n7238523.638081 |   0)   19354047 us |      } /* n_tty_read */\n7238523.638082 |   0)               |      tty_ldisc_deref() {\n7238523.638082 |   0)   0.064 us    |        ldsem_up_read();\n7238523.638082 |   0)   0.554 us    |      }\n7238523.638083 |   0)   0.074 us    |      get_seconds();\n7238523.638083 |   0)   19354052 us |    } /* tty_read */\n7238523.638084 |   0)   0.352 us    |    __fsnotify_parent();\n7238523.638085 |   0)   0.178 us    |    fsnotify();\n7238523.638085 |   0)   19354058 us |  } /* vfs_read */\n7238523.638156 |   0)               |  vfs_read() {\n7238523.638157 |   0)               |    rw_verify_area() {\n7238523.638157 |   0)               |      security_file_permission() {\n7238523.638158 |   0)               |        apparmor_file_permission() {\n7238523.638158 |   0)   0.183 us    |          common_file_perm();\n7238523.638159 |   0)   0.778 us    |        }\n7238523.638159 |   0)   0.081 us    |        __fsnotify_parent();\n7238523.638160 |   0)   0.104 us    |        fsnotify();\n7238523.638160 |   0)   2.662 us    |      }\n7238523.638161 |   0)   3.337 us    |    }\n7238523.638161 |   0)               |    tty_read() {\n7238523.638161 |   0)   0.067 us    |      tty_paranoia_check();\n7238523.638162 |   0)               |      tty_ldisc_ref_wait() {\n7238523.638162 |   0)   0.080 us    |        } /* ldsem_down_read */\n7238523.638163 |   0)   0.637 us    |      }\n7238523.638163 |   0)               |      n_tty_read() {\n7238523.638164 |   0)   0.078 us    |        _raw_spin_lock_irq();\n7238523.638164 |   0)   0.090 us    |        mutex_lock_interruptible();\n7238523.638165 |   0)   0.078 us    |        down_read();\n7238523.638165 |   0)               |        add_wait_queue() {\n7238523.638166 |   0)   0.070 us    |          _raw_spin_lock_irqsave();\n7238523.638166 |   0)   0.084 us    |          _raw_spin_unlock_irqrestore();\n7238523.638167 |   0)   1.111 us    |        }\n7238523.638167 |   0)   0.083 us    |        tty_hung_up_p();\n7238523.638168 |   0)   0.080 us    |        n_tty_set_room();\n7238523.638169 |   0)   0.068 us    |        up_read();\n7238523.638169 |   0)               |        schedule_timeout() {\n7238523.638170 |   0)               |          schedule() {\n7238523.638170 |   0)               |            __schedule() {\n7238523.638171 |   0)   0.078 us    |              rcu_note_context_switch();\n7238523.638171 |   0)   0.081 us    |              _raw_spin_lock_irq();\n7238523.638172 |   0)               |              deactivate_task() {\n7238523.638172 |   0)               |                dequeue_task() {\n7238523.638172 |   0)   0.181 us    |                  update_rq_clock();\n7238523.638173 |   0)               |                  dequeue_task_fair() {\n7238523.638174 |   0)               |                    dequeue_entity() {\n7238523.638174 |   0)               |                      update_curr() {\n7238523.638174 |   0)   0.257 us    |                        cpuacct_charge();\n7238523.638175 |   0)   0.982 us    |                      }\n7238523.638175 |   0)   0.079 us    |                      update_cfs_rq_blocked_load();\n7238523.638176 |   0)   0.080 us    |                      clear_buddies();\n7238523.638177 |   0)   0.096 us    |                      account_entity_dequeue();\n7238523.638177 |   0)               |                      update_cfs_shares() {\n7238523.638178 |   0)   0.113 us    |                        update_curr();\n7238523.638178 |   0)   0.087 us    |                        account_entity_dequeue();\n7238523.638179 |   0)   0.073 us    |                        account_entity_enqueue();\n7238523.638179 |   0)   1.948 us    |                      }\n7238523.638180 |   0)   5.913 us    |                    }\n7238523.638180 |   0)               |                    dequeue_entity() {\n7238523.638180 |   0)   0.086 us    |                      update_curr();\n7238523.638181 |   0)   0.079 us    |                      update_cfs_rq_blocked_load();\n7238523.638182 |   0)   0.076 us    |                      clear_buddies();\n7238523.638182 |   0)   0.076 us    |                      account_entity_dequeue();\n7238523.638183 |   0)   0.104 us    |                      update_cfs_shares();\n7238523.638183 |   0)   3.171 us    |                    }\n7238523.638184 |   0)   0.076 us    |                    hrtick_update();\n7238523.638184 |   0)   10.785 us   |                  }\n7238523.638184 |   0)   12.057 us   |                }\n7238523.638185 |   0)   12.704 us   |              }\n7238523.638185 |   0)               |              pick_next_task_fair() {\n7238523.638185 |   0)   0.074 us    |                check_cfs_rq_runtime();\n7238523.638186 |   0)               |                pick_next_entity() {\n7238523.638186 |   0)   0.067 us    |                  clear_buddies();\n7238523.638187 |   0)   0.544 us    |                }\n7238523.638187 |   0)               |                put_prev_entity() {\n7238523.638187 |   0)   0.079 us    |                  check_cfs_rq_runtime();\n7238523.638188 |   0)   0.612 us    |                }\n7238523.638188 |   0)               |                put_prev_entity() {\n7238523.638188 |   0)   0.076 us    |                  check_cfs_rq_runtime();\n7238523.638189 |   0)   0.618 us    |                }\n7238523.638189 |   0)               |                set_next_entity() {\n7238523.638190 |   0)   0.078 us    |                  update_stats_wait_end();\n7238523.638190 |   0)   0.712 us    |                }\n7238523.638190 |   0)   5.023 us    |              }\n7238523.638191 |   0)   0.086 us    |              paravirt_start_context_switch();\n7238523.638192 |   0)   0.070 us    |              xen_read_cr0();\n7238523.638193 |   0)               |              xen_write_cr0() {\n7238523.638193 |   0)   0.085 us    |                paravirt_get_lazy_mode();\n7238523.638194 |   0)   0.085 us    |                __xen_mc_entry();\n7238523.638194 |   0)   0.077 us    |                paravirt_get_lazy_mode();\n7238523.638195 |   0)   1.822 us    |              }\n7238523.638195 |   0)               |              xen_load_sp0() {\n7238523.638195 |   0)   0.074 us    |                paravirt_get_lazy_mode();\n7238523.638196 |   0)   0.085 us    |                __xen_mc_entry();\n7238523.638196 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7238523.638197 |   0)   1.754 us    |              }\n7238523.638197 |   0)               |              xen_load_tls() {\n7238523.638198 |   0)   0.069 us    |                paravirt_get_lazy_mode();\n7238523.638198 |   0)   0.082 us    |                paravirt_get_lazy_mode();\n7238523.638199 |   0)   0.127 us    |                load_TLS_descriptor();\n7238523.638199 |   0)   0.080 us    |                load_TLS_descriptor();\n7238523.638200 |   0)   0.094 us    |                load_TLS_descriptor();\n7238523.638201 |   0)   0.081 us    |                paravirt_get_lazy_mode();\n7238523.638202 |   0)   4.155 us    |              }\n7238523.638202 |   0)               |              xen_end_context_switch() {\n7238523.638202 |   0)   0.699 us    |                xen_mc_flush();\n7238523.638204 |   0)   0.089 us    |                paravirt_end_context_switch();\n7238523.638204 |   0)   1.915 us    |              }\n7238523.797630 |   0)               |              finish_task_switch() {\n7238523.797634 |   0)               |                xen_evtchn_do_upcall() {\n7238523.797634 |   0)               |                  irq_enter() {\n7238523.797634 |   0)   0.134 us    |                    rcu_irq_enter();\n7238523.797635 |   0)   0.688 us    |                  }\n7238523.797635 |   0)   0.055 us    |                  exit_idle();\n7238523.797635 |   0)               |                  __xen_evtchn_do_upcall() {\n7238523.797636 |   0)               |                    evtchn_2l_handle_events() {\n7238523.797636 |   0)   0.048 us    |                      irq_from_virq();\n7238523.797636 |   0)               |                      evtchn_from_irq() {\n7238523.797636 |   0)               |                        irq_get_irq_data() {\n7238523.797637 |   0)   0.061 us    |                          irq_to_desc();\n7238523.797637 |   0)   0.564 us    |                        }\n7238523.797637 |   0)   0.954 us    |                      }\n7238523.797638 |   0)               |                      get_evtchn_to_irq() {\n7238523.797638 |   0)   0.057 us    |                        evtchn_2l_max_channels();\n7238523.797638 |   0)   0.409 us    |                      }\n7238523.797638 |   0)               |                      generic_handle_irq() {\n7238523.797638 |   0)   0.052 us    |                        irq_to_desc();\n7238523.797639 |   0)               |                        handle_percpu_irq() {\n7238523.797639 |   0)               |                          ack_dynirq() {\n7238523.797639 |   0)               |                            evtchn_from_irq() {\n7238523.797639 |   0)               |                              irq_get_irq_data() {\n7238523.797640 |   0)   0.057 us    |                                irq_to_desc();\n7238523.797640 |   0)   0.440 us    |                              }\n7238523.797640 |   0)   0.746 us    |                            }\n7238523.797640 |   0)   0.056 us    |                            irq_move_irq();\n7238523.797641 |   0)   0.058 us    |                            evtchn_2l_clear_pending();\n7238523.797641 |   0)   1.729 us    |                          }\n7238523.797641 |   0)               |                          handle_irq_event_percpu() {\n7238523.797641 |   0)               |                            xen_irq_work_interrupt() {\n7238523.797642 |   0)               |                              irq_enter() {\n7238523.797642 |   0)   0.053 us    |                                rcu_irq_enter();\n7238523.797642 |   0)   0.396 us    |                              }\n7238523.797642 |   0)               |                              __wake_up() {\n7238523.797643 |   0)   0.053 us    |                                _raw_spin_lock_irqsave();\n7238523.797643 |   0)               |                                __wake_up_common() {\n7238523.797643 |   0)               |                                  autoremove_wake_function() {\n7238523.797644 |   0)               |                                    default_wake_function() {\n7238523.797644 |   0)               |                                      try_to_wake_up() {\n7238523.797644 |   0)   0.228 us    |                                        _raw_spin_lock_irqsave();\n7238523.797645 |   0)   0.194 us    |                                        task_waking_fair();\n7238523.797645 |   0)               |                                        select_task_rq_fair() {\n7238523.797645 |   0)   0.051 us    |                                          source_load();\n7238523.797646 |   0)   0.050 us    |                                          target_load();\n7238523.797646 |   0)   0.067 us    |                                          idle_cpu();\n7238523.797647 |   0)   0.050 us    |                                          cpus_share_cache();\n7238523.797647 |   0)   0.068 us    |                                          idle_cpu();\n7238523.797647 |   0)   1.983 us    |                                        }\n7238523.797648 |   0)   0.051 us    |                                        _raw_spin_lock();\n7238523.797648 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7238523.797648 |   0)               |                                          activate_task() {\n7238523.797648 |   0)               |                                            enqueue_task() {\n7238523.797648 |   0)   0.135 us    |                                              update_rq_clock();\n7238523.797649 |   0)               |                                              enqueue_task_fair() {\n7238523.797649 |   0)               |                                                enqueue_entity() {\n7238523.797649 |   0)   0.059 us    |                                                  update_curr();\n7238523.797650 |   0)   0.073 us    |                                                  __compute_runnable_contrib.part.51();\n7238523.797650 |   0)   0.066 us    |                                                  __update_entity_load_avg_contrib();\n7238523.797650 |   0)   0.059 us    |                                                  update_cfs_rq_blocked_load();\n7238523.797651 |   0)   0.064 us    |                                                  account_entity_enqueue();\n7238523.797651 |   0)   0.137 us    |                                                  update_cfs_shares();\n7238523.797651 |   0)   0.054 us    |                                                  place_entity();\n7238523.797652 |   0)   0.074 us    |                                                  __enqueue_entity();\n7238523.797652 |   0)   3.085 us    |                                                }\n7238523.797652 |   0)               |                                                enqueue_entity() {\n7238523.797653 |   0)   0.058 us    |                                                  update_curr();\n7238523.797654 |   0)   0.049 us    |                                                  update_cfs_rq_blocked_load();\n7238523.797654 |   0)   0.057 us    |                                                  account_entity_enqueue();\n7238523.797655 |   0)   0.066 us    |                                                  update_cfs_shares();\n7238523.797655 |   0)   0.049 us    |                                                  place_entity();\n7238523.797655 |   0)   0.051 us    |                                                  __enqueue_entity();\n7238523.797656 |   0)   3.432 us    |                                                }\n7238523.797656 |   0)   0.049 us    |                                                hrtick_update();\n7238523.797657 |   0)   7.552 us    |                                              }\n7238523.797657 |   0)   8.414 us    |                                            }\n7238523.797657 |   0)   8.753 us    |                                          }\n7238523.797657 |   0)               |                                          ttwu_do_wakeup() {\n7238523.797657 |   0)               |                                            check_preempt_curr() {\n7238523.797657 |   0)               |                                              resched_task() {\n7238523.797658 |   0)               |                                                xen_smp_send_reschedule() {\n7238523.797658 |   0)               |                                                  xen_send_IPI_one() {\n7238523.797658 |   0)               |                                                    notify_remote_via_irq() {\n7238523.797658 |   0)               |                                                      evtchn_from_irq() {\n7238523.797658 |   0)               |                                                        irq_get_irq_data() {\n7238523.797659 |   0)   0.069 us    |                                                          irq_to_desc();\n7238523.797659 |   0)   0.504 us    |                                                        }\n7238523.797659 |   0)   0.869 us    |                                                      }\n7238523.797660 |   0)   1.940 us    |                                                    } /* notify_remote_via_irq */\n7238523.797660 |   0)   2.319 us    |                                                  }\n7238523.797660 |   0)   2.712 us    |                                                }\n7238523.797661 |   0)   3.147 us    |                                              }\n7238523.797661 |   0)   3.625 us    |                                            }\n7238523.797662 |   0)   4.525 us    |                                          }\n7238523.797662 |   0)   13.961 us   |                                        }\n7238523.797662 |   0)   0.069 us    |                                        _raw_spin_unlock();\n7238523.797663 |   0)   0.168 us    |                                        ttwu_stat();\n7238523.797663 |   0)   0.076 us    |                                        _raw_spin_unlock_irqrestore();\n7238523.797664 |   0)   19.821 us   |                                      }\n7238523.797664 |   0)   20.301 us   |                                    }\n7238523.797664 |   0)   20.796 us   |                                  }\n7238523.797664 |   0)   21.367 us   |                                }\n7238523.797665 |   0)   0.071 us    |                                _raw_spin_unlock_irqrestore();\n7238523.797665 |   0)   22.621 us   |                              }\n7238523.797666 |   0)               |                              irq_exit() {\n7238523.797666 |   0)   0.085 us    |                                idle_cpu();\n7238523.797666 |   0)   0.106 us    |                                rcu_irq_exit();\n7238523.797667 |   0)   1.220 us    |                              }\n7238523.797667 |   0)   25.712 us   |                            }\n7238523.797668 |   0)   0.138 us    |                            add_interrupt_randomness();\n7238523.797668 |   0)   0.092 us    |                            note_interrupt();\n7238523.797669 |   0)   27.713 us   |                          }\n7238523.797669 |   0)   30.163 us   |                        }\n7238523.797669 |   0)   31.017 us   |                      }\n7238523.797670 |   0)   33.953 us   |                    }\n7238523.797670 |   0)   34.384 us   |                  }\n7238523.797670 |   0)               |                  irq_exit() {\n7238523.797671 |   0)   0.079 us    |                    idle_cpu();\n7238523.797671 |   0)   0.072 us    |                    rcu_irq_exit();\n7238523.797672 |   0)   1.023 us    |                  }\n7238523.797672 |   0)   37.789 us   |                }\n7238523.797672 |   0)   39.298 us   |              }\n7238523.797673 |   0)   159502.1 us |            }\n7238523.797673 |   0)   159502.8 us |          }\n7238523.797673 |   0)   159503.5 us |        }\n7238523.797674 |   0)   0.112 us    |        down_read();\n7238523.797675 |   0)               |        copy_from_read_buf() {\n7238523.797676 |   0)               |          tty_audit_add_data() {\n7238523.797676 |   0)   0.226 us    |            _raw_spin_lock_irqsave();\n7238523.797677 |   0)   0.075 us    |            _raw_spin_unlock_irqrestore();\n7238523.797677 |   0)   0.101 us    |            _raw_spin_lock_irqsave();\n7238523.797678 |   0)   0.068 us    |            _raw_spin_unlock_irqrestore();\n7238523.797679 |   0)   2.656 us    |          }\n7238523.797679 |   0)   3.762 us    |        }\n7238523.797679 |   0)   0.145 us    |        copy_from_read_buf();\n7238523.797680 |   0)   0.068 us    |        n_tty_set_room();\n7238523.797680 |   0)   0.058 us    |        n_tty_write_wakeup();\n7238523.797681 |   0)               |        __wake_up() {\n7238523.797682 |   0)   0.060 us    |          _raw_spin_lock_irqsave();\n7238523.797682 |   0)               |          __wake_up_common() {\n7238523.797683 |   0)   0.083 us    |            pollwake();\n7238523.797683 |   0)   0.739 us    |          }\n7238523.797683 |   0)   0.069 us    |          _raw_spin_unlock_irqrestore();\n7238523.797684 |   0)   2.745 us    |        }\n7238523.797684 |   0)   0.061 us    |        n_tty_set_room();\n7238523.797685 |   0)   0.074 us    |        up_read();\n7238523.797685 |   0)               |        remove_wait_queue() {\n7238523.797685 |   0)   0.075 us    |          _raw_spin_lock_irqsave();\n7238523.797686 |   0)   0.070 us    |          _raw_spin_unlock_irqrestore();\n7238523.797686 |   0)   1.110 us    |        }\n7238523.797687 |   0)   0.146 us    |        mutex_unlock();\n7238523.797687 |   0)   159524.0 us |      }\n7238523.797688 |   0)               |      tty_ldisc_deref() {\n7238523.797688 |   0)   0.070 us    |        ldsem_up_read();\n7238523.797689 |   0)   0.739 us    |      }\n7238523.797689 |   0)   0.066 us    |      get_seconds();\n7238523.797690 |   0)   159528.3 us |    }\n7238523.797690 |   0)   0.298 us    |    __fsnotify_parent();\n7238523.797691 |   0)   0.179 us    |    fsnotify();\n7238523.797692 |   0)   159534.6 us |  }\n7238523.797762 |   0)               |  vfs_read() {\n7238523.797763 |   0)               |    rw_verify_area() {\n7238523.797763 |   0)               |      security_file_permission() {\n7238523.797764 |   0)               |        apparmor_file_permission() {\n7238523.797764 |   0)   0.165 us    |          common_file_perm();\n7238523.797765 |   0)   0.732 us    |        }\n7238523.797765 |   0)   0.081 us    |        __fsnotify_parent();\n7238523.797766 |   0)   0.094 us    |        fsnotify();\n7238523.797766 |   0)   2.711 us    |      }\n7238523.797767 |   0)   3.386 us    |    }\n7238523.797767 |   0)               |    tty_read() {\n7238523.797767 |   0)   0.077 us    |      tty_paranoia_check();\n7238523.797768 |   0)               |      tty_ldisc_ref_wait() {\n7238523.797768 |   0)   0.083 us    |        ldsem_down_read();\n7238523.797769 |   0)   0.686 us    |      }\n7238523.797769 |   0)               |      n_tty_read() {\n7238523.797770 |   0)   0.071 us    |        _raw_spin_lock_irq();\n7238523.797770 |   0)   0.111 us    |        mutex_lock_interruptible();\n7238523.797771 |   0)   0.072 us    |        down_read();\n7238523.797771 |   0)               |        add_wait_queue() {\n7238523.797772 |   0)   0.083 us    |          _raw_spin_lock_irqsave();\n7238523.797772 |   0)   0.085 us    |          _raw_spin_unlock_irqrestore();\n7238523.797773 |   0)   1.124 us    |        }\n7238523.797773 |   0)   0.066 us    |        tty_hung_up_p();\n7238523.797774 |   0)   0.090 us    |        n_tty_set_room();\n7238523.797774 |   0)   0.064 us    |        up_read();\n7238523.797775 |   0)               |        schedule_timeout() {\n7238523.797775 |   0)               |          schedule() {\n7238523.797775 |   0)               |            __schedule() {\n7238523.797776 |   0)   0.083 us    |              rcu_note_context_switch();\n7238523.797776 |   0)   0.078 us    |              _raw_spin_lock_irq();\n7238523.797777 |   0)               |              deactivate_task() {\n7238523.797777 |   0)               |                dequeue_task() {\n7238523.797777 |   0)   0.191 us    |                  update_rq_clock();\n7238523.797778 |   0)               |                  dequeue_task_fair() {\n7238523.797778 |   0)               |                    dequeue_entity() {\n7238523.797779 |   0)               |                      update_curr() {\n7238523.797779 |   0)   0.179 us    |                        cpuacct_charge();\n7238523.797780 |   0)   0.902 us    |                      }\n7238523.797780 |   0)   0.070 us    |                      __update_entity_load_avg_contrib();\n7238523.797781 |   0)   0.152 us    |                      update_cfs_rq_blocked_load();\n7238523.797781 |   0)   0.073 us    |                      clear_buddies();\n7238523.797782 |   0)   0.074 us    |                      account_entity_dequeue();\n7238523.797783 |   0)               |                      update_cfs_shares() {\n7238523.797783 |   0)   0.111 us    |                        update_curr();\n7238523.797783 |   0)   0.082 us    |                        account_entity_dequeue();\n7238523.797784 |   0)   0.081 us    |                        account_entity_enqueue();\n7238523.797785 |   0)   2.330 us    |                      }\n7238523.797785 |   0)   6.633 us    |                    } /* dequeue_entity */\n7238523.797786 |   0)               |                    dequeue_entity() {\n7238523.797786 |   0)   0.078 us    |                      update_curr();\n7238523.797787 |   0)   0.086 us    |                      update_cfs_rq_blocked_load();\n7238523.797787 |   0)   0.076 us    |                      clear_buddies();\n7238523.797788 |   0)   0.079 us    |                      account_entity_dequeue();\n7238523.797789 |   0)   0.074 us    |                      update_cfs_shares();\n7238523.797789 |   0)   3.287 us    |                    }\n7238523.797789 |   0)   0.074 us    |                    hrtick_update();\n7238523.797790 |   0)   11.606 us   |                  }\n7238523.797790 |   0)   12.879 us   |                }\n7238523.797790 |   0)   13.406 us   |              }\n7238523.797791 |   0)               |              pick_next_task_fair() {\n7238523.797791 |   0)   0.073 us    |                check_cfs_rq_runtime();\n7238523.797792 |   0)               |                pick_next_entity() {\n7238523.797792 |   0)   0.076 us    |                  clear_buddies();\n7238523.797793 |   0)   0.663 us    |                }\n7238523.797793 |   0)               |                put_prev_entity() {\n7238523.797793 |   0)   0.076 us    |                  check_cfs_rq_runtime();\n7238523.797794 |   0)   0.598 us    |                }\n7238523.797794 |   0)               |                put_prev_entity() {\n7238523.797794 |   0)   0.078 us    |                  check_cfs_rq_runtime();\n7238523.797795 |   0)   0.618 us    |                }\n7238523.797795 |   0)               |                set_next_entity() {\n7238523.797795 |   0)   0.096 us    |                  update_stats_wait_end();\n7238523.797796 |   0)   0.738 us    |                }\n7238523.797796 |   0)   5.222 us    |              }\n7238523.797797 |   0)   0.078 us    |              paravirt_start_context_switch();\n7238523.797798 |   0)   0.071 us    |              xen_read_cr0();\n7238523.797799 |   0)               |              xen_write_cr0() {\n7238523.797799 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7238523.797800 |   0)   0.084 us    |                __xen_mc_entry();\n7238523.797800 |   0)   0.076 us    |                paravirt_get_lazy_mode();\n7238523.797801 |   0)   1.798 us    |              }\n7238523.797801 |   0)               |              xen_load_sp0() {\n7238523.797801 |   0)   0.080 us    |                paravirt_get_lazy_mode();\n7238523.797802 |   0)   0.076 us    |                __xen_mc_entry();\n7238523.797802 |   0)   0.073 us    |                paravirt_get_lazy_mode();\n7238523.797803 |   0)   1.623 us    |              }\n7238523.797803 |   0)               |              xen_load_tls() {\n7238523.797803 |   0)   0.082 us    |                paravirt_get_lazy_mode();\n7238523.797804 |   0)   0.084 us    |                paravirt_get_lazy_mode();\n7238523.797804 |   0)   0.136 us    |                load_TLS_descriptor();\n7238523.797805 |   0)   0.072 us    |                load_TLS_descriptor();\n7238523.797806 |   0)   0.080 us    |                load_TLS_descriptor();\n7238523.797806 |   0)   0.088 us    |                paravirt_get_lazy_mode();\n7238523.797807 |   0)   3.360 us    |              }\n7238523.797807 |   0)               |              xen_end_context_switch() {\n7238523.797807 |   0)   0.601 us    |                xen_mc_flush();\n7238523.797808 |   0)   0.098 us    |                paravirt_end_context_switch();\n7238523.797809 |   0)   1.902 us    |              }\n7238524.005649 |   0)               |              finish_task_switch() {\n7238524.005653 |   0)               |                xen_evtchn_do_upcall() {\n7238524.005653 |   0)               |                  irq_enter() {\n7238524.005653 |   0)   0.138 us    |                    rcu_irq_enter();\n7238524.005654 |   0)   0.753 us    |                  }\n7238524.005654 |   0)   0.056 us    |                  exit_idle();\n7238524.005655 |   0)               |                  __xen_evtchn_do_upcall() {\n7238524.005655 |   0)               |                    evtchn_2l_handle_events() {\n7238524.005655 |   0)   0.057 us    |                      irq_from_virq();\n7238524.005656 |   0)               |                      evtchn_from_irq() {\n7238524.005656 |   0)               |                        irq_get_irq_data() {\n7238524.005656 |   0)   0.050 us    |                          irq_to_desc();\n7238524.005656 |   0)   0.499 us    |                        }\n7238524.005657 |   0)   0.958 us    |                      }\n7238524.005657 |   0)               |                      get_evtchn_to_irq() {\n7238524.005657 |   0)   0.057 us    |                        evtchn_2l_max_channels();\n7238524.005658 |   0)   0.400 us    |                      }\n7238524.005659 |   0)               |                      generic_handle_irq() {\n7238524.005659 |   0)   0.052 us    |                        irq_to_desc();\n7238524.005659 |   0)               |                        handle_percpu_irq() {\n7238524.005659 |   0)               |                          ack_dynirq() {\n7238524.005659 |   0)               |                            evtchn_from_irq() {\n7238524.005660 |   0)               |                              irq_get_irq_data() {\n7238524.005660 |   0)   0.056 us    |                                irq_to_desc();\n7238524.005660 |   0)   0.439 us    |                              }\n7238524.005660 |   0)   0.739 us    |                            }\n7238524.005661 |   0)   0.051 us    |                            irq_move_irq();\n7238524.005661 |   0)   0.051 us    |                            evtchn_2l_clear_pending();\n7238524.005661 |   0)   1.963 us    |                          }\n7238524.005662 |   0)               |                          handle_irq_event_percpu() {\n7238524.005662 |   0)               |                            xen_irq_work_interrupt() {\n7238524.005662 |   0)               |                              irq_enter() {\n7238524.005662 |   0)   0.053 us    |                                rcu_irq_enter();\n7238524.005663 |   0)   0.392 us    |                              }\n7238524.005663 |   0)               |                              __wake_up() {\n7238524.005663 |   0)   0.058 us    |                                _raw_spin_lock_irqsave();\n7238524.005664 |   0)               |                                __wake_up_common() {\n7238524.005664 |   0)               |                                  autoremove_wake_function() {\n7238524.005664 |   0)               |                                    default_wake_function() {\n7238524.005665 |   0)               |                                      try_to_wake_up() {\n7238524.005665 |   0)   0.226 us    |                                        _raw_spin_lock_irqsave();\n7238524.005665 |   0)   0.392 us    |                                        task_waking_fair();\n7238524.005666 |   0)               |                                        select_task_rq_fair() {\n7238524.005666 |   0)   0.067 us    |                                          source_load();\n7238524.005667 |   0)   0.057 us    |                                          target_load();\n7238524.005667 |   0)   0.065 us    |                                          idle_cpu();\n7238524.005668 |   0)   0.050 us    |                                          cpus_share_cache();\n7238524.005668 |   0)   0.080 us    |                                          idle_cpu();\n7238524.005668 |   0)   2.053 us    |                                        }\n7238524.005669 |   0)   0.051 us    |                                        _raw_spin_lock();\n7238524.005669 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7238524.005669 |   0)               |                                          activate_task() {\n7238524.005669 |   0)               |                                            enqueue_task() {\n7238524.005669 |   0)   0.165 us    |                                              update_rq_clock();\n7238524.005670 |   0)               |                                              enqueue_task_fair() {\n7238524.005670 |   0)               |                                                enqueue_entity() {\n7238524.005670 |   0)   0.065 us    |                                                  update_curr();\n7238524.005671 |   0)   0.078 us    |                                                  __compute_runnable_contrib.part.51();\n7238524.005671 |   0)   0.070 us    |                                                  __update_entity_load_avg_contrib();\n7238524.005671 |   0)   0.051 us    |                                                  update_cfs_rq_blocked_load();\n7238524.005672 |   0)   0.069 us    |                                                  account_entity_enqueue();\n7238524.005672 |   0)   0.132 us    |                                                  update_cfs_shares();\n7238524.005673 |   0)   0.054 us    |                                                  place_entity();\n7238524.005673 |   0)   0.081 us    |                                                  __enqueue_entity();\n7238524.005673 |   0)   3.111 us    |                                                }\n7238524.005673 |   0)               |                                                enqueue_entity() {\n7238524.005674 |   0)   0.059 us    |                                                  update_curr();\n7238524.005674 |   0)   0.057 us    |                                                  update_cfs_rq_blocked_load();\n7238524.005674 |   0)   0.067 us    |                                                  account_entity_enqueue();\n7238524.005675 |   0)   0.082 us    |                                                  update_cfs_shares();\n7238524.005675 |   0)   0.120 us    |                                                  place_entity();\n7238524.005675 |   0)   0.051 us    |                                                  __enqueue_entity();\n7238524.005676 |   0)   2.075 us    |                                                }\n7238524.005676 |   0)   0.049 us    |                                                hrtick_update();\n7238524.005676 |   0)   6.167 us    |                                              }\n7238524.005676 |   0)   6.979 us    |                                            }\n7238524.005676 |   0)   7.317 us    |                                          }\n7238524.005677 |   0)               |                                          ttwu_do_wakeup() {\n7238524.005677 |   0)               |                                            check_preempt_curr() {\n7238524.005677 |   0)               |                                              resched_task() {\n7238524.005677 |   0)               |                                                xen_smp_send_reschedule() {\n7238524.005677 |   0)               |                                                  xen_send_IPI_one() {\n7238524.005678 |   0)               |                                                    notify_remote_via_irq() {\n7238524.005678 |   0)               |                                                      evtchn_from_irq() {\n7238524.005678 |   0)               |                                                        irq_get_irq_data() {\n7238524.005678 |   0)   0.051 us    |                                                          irq_to_desc();\n7238524.005679 |   0)   0.545 us    |                                                        }\n7238524.005679 |   0)   0.910 us    |                                                      }\n7238524.005680 |   0)   1.962 us    |                                                    } /* notify_remote_via_irq */\n7238524.005680 |   0)   2.332 us    |                                                  }\n7238524.005680 |   0)   2.684 us    |                                                }\n7238524.005681 |   0)   3.606 us    |                                              }\n7238524.005681 |   0)   4.064 us    |                                            }\n7238524.005682 |   0)   5.129 us    |                                          }\n7238524.005682 |   0)   13.194 us   |                                        }\n7238524.005683 |   0)   0.066 us    |                                        _raw_spin_unlock();\n7238524.005683 |   0)   0.165 us    |                                        ttwu_stat();\n7238524.005684 |   0)   0.070 us    |                                        _raw_spin_unlock_irqrestore();\n7238524.005684 |   0)   19.634 us   |                                      }\n7238524.005685 |   0)   20.080 us   |                                    }\n7238524.005685 |   0)   20.608 us   |                                  }\n7238524.005685 |   0)   21.348 us   |                                }\n7238524.005685 |   0)   0.084 us    |                                _raw_spin_unlock_irqrestore();\n7238524.005686 |   0)   22.728 us   |                              }\n7238524.005686 |   0)               |                              irq_exit() {\n7238524.005687 |   0)   0.077 us    |                                idle_cpu();\n7238524.005687 |   0)   0.093 us    |                                rcu_irq_exit();\n7238524.005688 |   0)   1.101 us    |                              }\n7238524.005688 |   0)   25.644 us   |                            }\n7238524.005688 |   0)   0.138 us    |                            add_interrupt_randomness();\n7238524.005689 |   0)   0.083 us    |                            note_interrupt();\n7238524.005689 |   0)   27.672 us   |                          }\n7238524.005690 |   0)   30.410 us   |                        }\n7238524.005690 |   0)   31.458 us   |                      }\n7238524.005690 |   0)   35.276 us   |                    }\n7238524.005691 |   0)   35.797 us   |                  }\n7238524.005691 |   0)               |                  irq_exit() {\n7238524.005691 |   0)   0.066 us    |                    idle_cpu();\n7238524.005692 |   0)   0.080 us    |                    rcu_irq_exit();\n7238524.005692 |   0)   1.110 us    |                  }\n7238524.005693 |   0)   39.440 us   |                }\n7238524.005693 |   0)   41.142 us   |              }\n7238524.005694 |   0)   207918.1 us |            }\n7238524.005694 |   0)   207918.7 us |          }\n7238524.005694 |   0)   207919.4 us |        }\n7238524.005695 |   0)   0.068 us    |        down_read();\n7238524.005696 |   0)               |        copy_from_read_buf() {\n7238524.005697 |   0)               |          tty_audit_add_data() {\n7238524.005697 |   0)   0.233 us    |            _raw_spin_lock_irqsave();\n7238524.005698 |   0)   0.076 us    |            _raw_spin_unlock_irqrestore();\n7238524.005699 |   0)   0.076 us    |            _raw_spin_lock_irqsave();\n7238524.005699 |   0)   0.078 us    |            _raw_spin_unlock_irqrestore();\n7238524.005700 |   0)   2.696 us    |          }\n7238524.005700 |   0)   4.335 us    |        }\n7238524.005701 |   0)   0.086 us    |        copy_from_read_buf();\n7238524.005701 |   0)   0.074 us    |        n_tty_set_room();\n7238524.005702 |   0)   0.085 us    |        n_tty_write_wakeup();\n7238524.005702 |   0)               |        __wake_up() {\n7238524.005703 |   0)   0.061 us    |          _raw_spin_lock_irqsave();\n7238524.005703 |   0)               |          __wake_up_common() {\n7238524.005703 |   0)   0.080 us    |            pollwake();\n7238524.005704 |   0)   0.687 us    |          }\n7238524.005704 |   0)   0.063 us    |          _raw_spin_unlock_irqrestore();\n7238524.005705 |   0)   2.040 us    |        }\n7238524.005705 |   0)   0.071 us    |        n_tty_set_room();\n7238524.005706 |   0)   0.074 us    |        up_read();\n7238524.005706 |   0)               |        remove_wait_queue() {\n7238524.005706 |   0)   0.074 us    |          _raw_spin_lock_irqsave();\n7238524.005707 |   0)   0.069 us    |          _raw_spin_unlock_irqrestore();\n7238524.005707 |   0)   1.076 us    |        }\n7238524.005708 |   0)   0.139 us    |        mutex_unlock();\n7238524.005708 |   0)   207939.0 us |      }\n7238524.005709 |   0)               |      tty_ldisc_deref() {\n7238524.005709 |   0)   0.077 us    |        ldsem_up_read();\n7238524.005710 |   0)   0.702 us    |      }\n7238524.005710 |   0)   0.068 us    |      get_seconds();\n7238524.005711 |   0)   207943.4 us |    }\n7238524.005712 |   0)   0.301 us    |    __fsnotify_parent();\n7238524.005713 |   0)   0.157 us    |    fsnotify();\n7238524.005713 |   0)   207950.3 us |  }\n7238524.005783 |   0)               |  vfs_read() {\n7238524.005784 |   0)               |    rw_verify_area() {\n7238524.005784 |   0)               |      security_file_permission() {\n7238524.005785 |   0)               |        apparmor_file_permission() {\n7238524.005785 |   0)   0.164 us    |          common_file_perm();\n7238524.005786 |   0)   0.790 us    |        }\n7238524.005786 |   0)   0.080 us    |        __fsnotify_parent();\n7238524.005787 |   0)   0.094 us    |        fsnotify();\n7238524.005787 |   0)   2.683 us    |      }\n7238524.005788 |   0)   3.313 us    |    }\n7238524.005788 |   0)               |    tty_read() {\n7238524.005788 |   0)   0.087 us    |      tty_paranoia_check();\n7238524.005789 |   0)               |      tty_ldisc_ref_wait() {\n7238524.005789 |   0)   0.080 us    |        ldsem_down_read();\n7238524.005790 |   0)   0.683 us    |      }\n7238524.005790 |   0)               |      n_tty_read() {\n7238524.005791 |   0)   0.080 us    |        _raw_spin_lock_irq();\n7238524.005791 |   0)   0.104 us    |        mutex_lock_interruptible();\n7238524.005792 |   0)   0.070 us    |        down_read();\n7238524.005792 |   0)               |        add_wait_queue() {\n7238524.005793 |   0)   0.079 us    |          _raw_spin_lock_irqsave();\n7238524.005793 |   0)   0.087 us    |          _raw_spin_unlock_irqrestore();\n7238524.005794 |   0)   1.147 us    |        }\n7238524.005794 |   0)   0.078 us    |        tty_hung_up_p();\n7238524.005795 |   0)   0.071 us    |        n_tty_set_room();\n7238524.005795 |   0)   0.077 us    |        up_read();\n7238524.005796 |   0)               |        schedule_timeout() {\n7238524.005796 |   0)               |          schedule() {\n7238524.005796 |   0)               |            __schedule() {\n7238524.005797 |   0)   0.087 us    |              rcu_note_context_switch();\n7238524.005797 |   0)   0.075 us    |              _raw_spin_lock_irq();\n7238524.005798 |   0)               |              deactivate_task() {\n7238524.005798 |   0)               |                dequeue_task() {\n7238524.005798 |   0)   0.177 us    |                  update_rq_clock();\n7238524.005799 |   0)               |                  dequeue_task_fair() {\n7238524.005799 |   0)               |                    dequeue_entity() {\n7238524.005800 |   0)               |                      update_curr() {\n7238524.005800 |   0)   0.334 us    |                        cpuacct_charge();\n7238524.005801 |   0)   1.199 us    |                      }\n7238524.005802 |   0)   0.081 us    |                      __update_entity_load_avg_contrib();\n7238524.005802 |   0)   0.064 us    |                      update_cfs_rq_blocked_load();\n7238524.005803 |   0)   0.076 us    |                      clear_buddies();\n7238524.005803 |   0)   0.079 us    |                      account_entity_dequeue();\n7238524.005804 |   0)               |                      update_cfs_shares() {\n7238524.005804 |   0)   0.108 us    |                        update_curr();\n7238524.005805 |   0)   0.083 us    |                        account_entity_dequeue();\n7238524.005805 |   0)   0.084 us    |                        account_entity_enqueue();\n7238524.005806 |   0)   1.869 us    |                      }\n7238524.005806 |   0)   6.530 us    |                    } /* dequeue_entity */\n7238524.005807 |   0)               |                    dequeue_entity() {\n7238524.005807 |   0)   0.104 us    |                      update_curr();\n7238524.005808 |   0)   0.115 us    |                      __update_entity_load_avg_contrib();\n7238524.005808 |   0)   0.069 us    |                      update_cfs_rq_blocked_load();\n7238524.005809 |   0)   0.066 us    |                      clear_buddies();\n7238524.005809 |   0)   0.086 us    |                      account_entity_dequeue();\n7238524.005810 |   0)   0.102 us    |                      update_cfs_shares();\n7238524.005811 |   0)   3.907 us    |                    }\n7238524.005811 |   0)   0.071 us    |                    hrtick_update();\n7238524.005812 |   0)   12.301 us   |                  }\n7238524.005812 |   0)   13.546 us   |                }\n7238524.005812 |   0)   14.105 us   |              }\n7238524.005812 |   0)               |              pick_next_task_fair() {\n7238524.005813 |   0)   0.078 us    |                check_cfs_rq_runtime();\n7238524.005813 |   0)               |                pick_next_entity() {\n7238524.005814 |   0)   0.071 us    |                  clear_buddies();\n7238524.005815 |   0)   0.585 us    |                }\n7238524.005815 |   0)               |                put_prev_entity() {\n7238524.005815 |   0)   0.073 us    |                  check_cfs_rq_runtime();\n7238524.005816 |   0)   0.717 us    |                }\n7238524.005816 |   0)               |                put_prev_entity() {\n7238524.005817 |   0)   0.080 us    |                  check_cfs_rq_runtime();\n7238524.005817 |   0)   0.687 us    |                }\n7238524.005817 |   0)               |                set_next_entity() {\n7238524.005818 |   0)   0.091 us    |                  update_stats_wait_end();\n7238524.005818 |   0)   0.786 us    |                }\n7238524.005819 |   0)   6.135 us    |              }\n7238524.005820 |   0)   0.091 us    |              paravirt_start_context_switch();\n7238524.005821 |   0)   0.089 us    |              xen_read_cr0();\n7238524.005821 |   0)               |              xen_write_cr0() {\n7238524.005821 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7238524.005822 |   0)   0.083 us    |                __xen_mc_entry();\n7238524.005823 |   0)   0.074 us    |                paravirt_get_lazy_mode();\n7238524.005823 |   0)   1.657 us    |              }\n7238524.005823 |   0)               |              xen_load_sp0() {\n7238524.005824 |   0)   0.074 us    |                paravirt_get_lazy_mode();\n7238524.005824 |   0)   0.083 us    |                __xen_mc_entry();\n7238524.005825 |   0)   0.087 us    |                paravirt_get_lazy_mode();\n7238524.005825 |   0)   1.764 us    |              }\n7238524.005826 |   0)               |              xen_load_tls() {\n7238524.005826 |   0)   0.077 us    |                paravirt_get_lazy_mode();\n7238524.005826 |   0)   0.084 us    |                paravirt_get_lazy_mode();\n7238524.005827 |   0)   0.150 us    |                load_TLS_descriptor();\n7238524.005828 |   0)   0.082 us    |                load_TLS_descriptor();\n7238524.005828 |   0)   0.084 us    |                load_TLS_descriptor();\n7238524.005829 |   0)   0.076 us    |                paravirt_get_lazy_mode();\n7238524.005829 |   0)   3.388 us    |              }\n7238524.005829 |   0)               |              xen_end_context_switch() {\n7238524.005830 |   0)   0.731 us    |                xen_mc_flush();\n7238524.005831 |   0)   0.093 us    |                paravirt_end_context_switch();\n7238524.005831 |   0)   1.836 us    |              }\n7238524.141853 |   0)               |              finish_task_switch() {\n7238524.141857 |   0)               |                xen_evtchn_do_upcall() {\n7238524.141858 |   0)               |                  irq_enter() {\n7238524.141858 |   0)   0.133 us    |                    rcu_irq_enter();\n7238524.141859 |   0)   0.766 us    |                  }\n7238524.141859 |   0)   0.056 us    |                  exit_idle();\n7238524.141859 |   0)               |                  __xen_evtchn_do_upcall() {\n7238524.141859 |   0)               |                    evtchn_2l_handle_events() {\n7238524.141860 |   0)   0.049 us    |                      irq_from_virq();\n7238524.141860 |   0)               |                      evtchn_from_irq() {\n7238524.141860 |   0)               |                        irq_get_irq_data() {\n7238524.141860 |   0)   0.058 us    |                          irq_to_desc();\n7238524.141861 |   0)   0.498 us    |                        }\n7238524.141861 |   0)   0.897 us    |                      }\n7238524.141861 |   0)               |                      get_evtchn_to_irq() {\n7238524.141861 |   0)   0.049 us    |                        evtchn_2l_max_channels();\n7238524.141862 |   0)   0.392 us    |                      }\n7238524.141862 |   0)               |                      generic_handle_irq() {\n7238524.141862 |   0)   0.061 us    |                        irq_to_desc();\n7238524.141862 |   0)               |                        handle_percpu_irq() {\n7238524.141863 |   0)               |                          ack_dynirq() {\n7238524.141863 |   0)               |                            evtchn_from_irq() {\n7238524.141863 |   0)               |                              irq_get_irq_data() {\n7238524.141863 |   0)   0.051 us    |                                irq_to_desc();\n7238524.141863 |   0)   0.439 us    |                              }\n7238524.141864 |   0)   0.745 us    |                            }\n7238524.141864 |   0)   0.049 us    |                            irq_move_irq();\n7238524.141864 |   0)   0.060 us    |                            evtchn_2l_clear_pending();\n7238524.141864 |   0)   1.714 us    |                          }\n7238524.141865 |   0)               |                          handle_irq_event_percpu() {\n7238524.141865 |   0)               |                            xen_irq_work_interrupt() {\n7238524.141865 |   0)               |                              irq_enter() {\n7238524.141865 |   0)   0.053 us    |                                rcu_irq_enter();\n7238524.141866 |   0)   0.371 us    |                              }\n7238524.141866 |   0)               |                              __wake_up() {\n7238524.141866 |   0)   0.051 us    |                                _raw_spin_lock_irqsave();\n7238524.141867 |   0)               |                                __wake_up_common() {\n7238524.141867 |   0)               |                                  autoremove_wake_function() {\n7238524.141867 |   0)               |                                    default_wake_function() {\n7238524.141867 |   0)               |                                      try_to_wake_up() {\n7238524.141868 |   0)   0.213 us    |                                        _raw_spin_lock_irqsave();\n7238524.141869 |   0)   0.196 us    |                                        task_waking_fair();\n7238524.141870 |   0)               |                                        select_task_rq_fair() {\n7238524.141870 |   0)   0.051 us    |                                          source_load();\n7238524.141870 |   0)   0.049 us    |                                          target_load();\n7238524.141871 |   0)   0.065 us    |                                          idle_cpu();\n7238524.141871 |   0)   0.051 us    |                                          cpus_share_cache();\n7238524.141872 |   0)   0.078 us    |                                          idle_cpu();\n7238524.141872 |   0)   2.427 us    |                                        }\n7238524.141873 |   0)   0.050 us    |                                        _raw_spin_lock();\n7238524.141873 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7238524.141873 |   0)               |                                          activate_task() {\n7238524.141873 |   0)               |                                            enqueue_task() {\n7238524.141873 |   0)   0.170 us    |                                              update_rq_clock();\n7238524.141874 |   0)               |                                              enqueue_task_fair() {\n7238524.141874 |   0)               |                                                enqueue_entity() {\n7238524.141874 |   0)   0.058 us    |                                                  update_curr();\n7238524.141875 |   0)   0.076 us    |                                                  __compute_runnable_contrib.part.51();\n7238524.141875 |   0)   0.059 us    |                                                  __update_entity_load_avg_contrib();\n7238524.141875 |   0)   0.060 us    |                                                  update_cfs_rq_blocked_load();\n7238524.141876 |   0)   0.064 us    |                                                  account_entity_enqueue();\n7238524.141876 |   0)   0.123 us    |                                                  update_cfs_shares();\n7238524.141876 |   0)   0.055 us    |                                                  place_entity();\n7238524.141877 |   0)   0.078 us    |                                                  __enqueue_entity();\n7238524.141877 |   0)   3.039 us    |                                                }\n7238524.141877 |   0)               |                                                enqueue_entity() {\n7238524.141877 |   0)   0.065 us    |                                                  update_curr();\n7238524.141878 |   0)   0.050 us    |                                                  update_cfs_rq_blocked_load();\n7238524.141878 |   0)   0.049 us    |                                                  account_entity_enqueue();\n7238524.141878 |   0)   0.081 us    |                                                  update_cfs_shares();\n7238524.141879 |   0)   0.049 us    |                                                  place_entity();\n7238524.141879 |   0)   0.051 us    |                                                  __enqueue_entity();\n7238524.141879 |   0)   2.021 us    |                                                }\n7238524.141880 |   0)   0.049 us    |                                                hrtick_update();\n7238524.141880 |   0)   6.040 us    |                                              }\n7238524.141880 |   0)   6.874 us    |                                            }\n7238524.141880 |   0)   7.212 us    |                                          }\n7238524.141880 |   0)               |                                          ttwu_do_wakeup() {\n7238524.141881 |   0)               |                                            check_preempt_curr() {\n7238524.141881 |   0)               |                                              resched_task() {\n7238524.141881 |   0)               |                                                xen_smp_send_reschedule() {\n7238524.141881 |   0)               |                                                  xen_send_IPI_one() {\n7238524.141881 |   0)               |                                                    notify_remote_via_irq() {\n7238524.141881 |   0)               |                                                      evtchn_from_irq() {\n7238524.141882 |   0)               |                                                        irq_get_irq_data() {\n7238524.141882 |   0)   0.049 us    |                                                          irq_to_desc();\n7238524.141882 |   0)   0.497 us    |                                                        }\n7238524.141882 |   0)   0.860 us    |                                                      }\n7238524.141883 |   0)   1.882 us    |                                                    } /* notify_remote_via_irq */\n7238524.141884 |   0)   2.257 us    |                                                  }\n7238524.141884 |   0)   2.619 us    |                                                }\n7238524.141884 |   0)   3.079 us    |                                              }\n7238524.141884 |   0)   3.526 us    |                                            }\n7238524.141885 |   0)   4.485 us    |                                          }\n7238524.141885 |   0)   12.333 us   |                                        }\n7238524.141885 |   0)   0.062 us    |                                        _raw_spin_unlock();\n7238524.141886 |   0)   0.169 us    |                                        ttwu_stat();\n7238524.141887 |   0)   0.076 us    |                                        _raw_spin_unlock_irqrestore();\n7238524.141887 |   0)   19.434 us   |                                      }\n7238524.141887 |   0)   19.909 us   |                                    }\n7238524.141888 |   0)   20.377 us   |                                  }\n7238524.141888 |   0)   21.020 us   |                                }\n7238524.141888 |   0)   0.075 us    |                                _raw_spin_unlock_irqrestore();\n7238524.141888 |   0)   22.268 us   |                              }\n7238524.141889 |   0)               |                              irq_exit() {\n7238524.141889 |   0)   0.087 us    |                                idle_cpu();\n7238524.141889 |   0)   0.101 us    |                                rcu_irq_exit();\n7238524.141890 |   0)   1.127 us    |                              }\n7238524.141890 |   0)   25.163 us   |                            }\n7238524.141891 |   0)   0.133 us    |                            add_interrupt_randomness();\n7238524.141892 |   0)   0.083 us    |                            note_interrupt();\n7238524.141892 |   0)   27.453 us   |                          }\n7238524.141893 |   0)   30.024 us   |                        }\n7238524.141893 |   0)   30.898 us   |                      }\n7238524.141893 |   0)   33.683 us   |                    }\n7238524.141893 |   0)   34.097 us   |                  }\n7238524.141894 |   0)               |                  irq_exit() {\n7238524.141894 |   0)   0.065 us    |                    idle_cpu();\n7238524.141895 |   0)   0.076 us    |                    rcu_irq_exit();\n7238524.141895 |   0)   1.135 us    |                  }\n7238524.141895 |   0)   37.746 us   |                }\n7238524.141896 |   0)   39.634 us   |              }\n7238524.141897 |   0)   136100.0 us |            }\n7238524.141897 |   0)   136100.6 us |          }\n7238524.141897 |   0)   136101.4 us |        }\n7238524.141898 |   0)   0.093 us    |        down_read();\n7238524.141899 |   0)               |        copy_from_read_buf() {\n7238524.141900 |   0)               |          tty_audit_add_data() {\n7238524.141900 |   0)   0.238 us    |            _raw_spin_lock_irqsave();\n7238524.141901 |   0)   0.069 us    |            _raw_spin_unlock_irqrestore();\n7238524.141901 |   0)   0.090 us    |            _raw_spin_lock_irqsave();\n7238524.141902 |   0)   0.077 us    |            _raw_spin_unlock_irqrestore();\n7238524.141902 |   0)   2.513 us    |          }\n7238524.141903 |   0)   3.632 us    |        }\n7238524.141903 |   0)   0.085 us    |        copy_from_read_buf();\n7238524.141904 |   0)   0.066 us    |        n_tty_set_room();\n7238524.141905 |   0)   0.067 us    |        n_tty_write_wakeup();\n7238524.141905 |   0)               |        __wake_up() {\n7238524.141906 |   0)   0.070 us    |          _raw_spin_lock_irqsave();\n7238524.141906 |   0)               |          __wake_up_common() {\n7238524.141906 |   0)   0.086 us    |            pollwake();\n7238524.141907 |   0)   0.620 us    |          }\n7238524.141907 |   0)   0.064 us    |          _raw_spin_unlock_irqrestore();\n7238524.141907 |   0)   1.980 us    |        }\n7238524.141908 |   0)   0.059 us    |        n_tty_set_room();\n7238524.141909 |   0)   0.071 us    |        up_read();\n7238524.141909 |   0)               |        remove_wait_queue() {\n7238524.141909 |   0)   0.079 us    |          _raw_spin_lock_irqsave();\n7238524.141910 |   0)   0.082 us    |          _raw_spin_unlock_irqrestore();\n7238524.141910 |   0)   1.164 us    |        }\n7238524.141910 |   0)   0.142 us    |        mutex_unlock();\n7238524.141911 |   0)   136120.9 us |      }\n7238524.141911 |   0)               |      tty_ldisc_deref() {\n7238524.141912 |   0)   0.062 us    |        ldsem_up_read();\n7238524.141912 |   0)   0.593 us    |      }\n7238524.141912 |   0)   0.079 us    |      get_seconds();\n7238524.141913 |   0)   136125.1 us |    }\n7238524.141914 |   0)   0.280 us    |    __fsnotify_parent();\n7238524.141915 |   0)   0.187 us    |    fsnotify();\n7238524.141915 |   0)   136131.2 us |  }\n7238524.141988 |   0)               |  vfs_read() {\n7238524.141989 |   0)               |    rw_verify_area() {\n7238524.141989 |   0)               |      security_file_permission() {\n7238524.141989 |   0)               |        apparmor_file_permission() {\n7238524.141990 |   0)   0.149 us    |          common_file_perm();\n7238524.141990 |   0)   0.774 us    |        }\n7238524.141991 |   0)   0.079 us    |        __fsnotify_parent();\n7238524.141991 |   0)   0.095 us    |        fsnotify();\n7238524.141992 |   0)   2.558 us    |      }\n7238524.141992 |   0)   3.300 us    |    }\n7238524.141993 |   0)               |    tty_read() {\n7238524.141993 |   0)   0.076 us    |      tty_paranoia_check();\n7238524.141994 |   0)               |      tty_ldisc_ref_wait() {\n7238524.141994 |   0)   0.081 us    |        ldsem_down_read();\n7238524.141995 |   0)   0.689 us    |      }\n7238524.141995 |   0)               |      n_tty_read() {\n7238524.141995 |   0)   0.073 us    |        _raw_spin_lock_irq();\n7238524.141996 |   0)   0.110 us    |        mutex_lock_interruptible();\n7238524.141997 |   0)   0.069 us    |        down_read();\n7238524.141998 |   0)               |        add_wait_queue() {\n7238524.141998 |   0)   0.079 us    |          _raw_spin_lock_irqsave();\n7238524.141999 |   0)   0.078 us    |          _raw_spin_unlock_irqrestore();\n7238524.141999 |   0)   1.201 us    |        }\n7238524.142000 |   0)   0.067 us    |        tty_hung_up_p();\n7238524.142000 |   0)   0.078 us    |        n_tty_set_room();\n7238524.142001 |   0)   0.079 us    |        up_read();\n7238524.142001 |   0)               |        schedule_timeout() {\n7238524.142002 |   0)               |          schedule() {\n7238524.142002 |   0)               |            __schedule() {\n7238524.142002 |   0)   0.076 us    |              rcu_note_context_switch();\n7238524.142003 |   0)   0.080 us    |              _raw_spin_lock_irq();\n7238524.142004 |   0)               |              deactivate_task() {\n7238524.142004 |   0)               |                dequeue_task() {\n7238524.142004 |   0)   0.178 us    |                  update_rq_clock();\n7238524.142005 |   0)               |                  dequeue_task_fair() {\n7238524.142005 |   0)               |                    dequeue_entity() {\n7238524.142005 |   0)               |                      update_curr() {\n7238524.142006 |   0)   0.263 us    |                        cpuacct_charge();\n7238524.142007 |   0)   0.965 us    |                      }\n7238524.142007 |   0)   0.075 us    |                      update_cfs_rq_blocked_load();\n7238524.142008 |   0)   0.065 us    |                      clear_buddies();\n7238524.142008 |   0)   0.084 us    |                      account_entity_dequeue();\n7238524.142009 |   0)               |                      update_cfs_shares() {\n7238524.142009 |   0)   0.115 us    |                        update_curr();\n7238524.142010 |   0)   0.084 us    |                        account_entity_dequeue();\n7238524.142010 |   0)   0.068 us    |                        account_entity_enqueue();\n7238524.142011 |   0)   1.754 us    |                      }\n7238524.142011 |   0)   5.580 us    |                    }\n7238524.142012 |   0)               |                    dequeue_entity() {\n7238524.142012 |   0)   0.089 us    |                      update_curr();\n7238524.142012 |   0)   0.101 us    |                      update_cfs_rq_blocked_load();\n7238524.142013 |   0)   0.076 us    |                      clear_buddies();\n7238524.142013 |   0)   0.078 us    |                      account_entity_dequeue();\n7238524.142014 |   0)   0.076 us    |                      update_cfs_shares();\n7238524.142015 |   0)   3.071 us    |                    }\n7238524.142015 |   0)   0.078 us    |                    hrtick_update();\n7238524.142016 |   0)   10.525 us   |                  }\n7238524.142016 |   0)   11.803 us   |                }\n7238524.142016 |   0)   12.447 us   |              }\n7238524.142017 |   0)               |              pick_next_task_fair() {\n7238524.142017 |   0)   0.069 us    |                check_cfs_rq_runtime();\n7238524.142017 |   0)               |                pick_next_entity() {\n7238524.142018 |   0)   0.061 us    |                  clear_buddies();\n7238524.142018 |   0)   0.601 us    |                }\n7238524.142019 |   0)               |                put_prev_entity() {\n7238524.142019 |   0)   0.069 us    |                  check_cfs_rq_runtime();\n7238524.142019 |   0)   0.605 us    |                }\n7238524.142020 |   0)               |                put_prev_entity() {\n7238524.142020 |   0)   0.076 us    |                  check_cfs_rq_runtime();\n7238524.142020 |   0)   0.609 us    |                }\n7238524.142021 |   0)               |                set_next_entity() {\n7238524.142021 |   0)   0.088 us    |                  update_stats_wait_end();\n7238524.142022 |   0)   0.768 us    |                }\n7238524.142022 |   0)   5.183 us    |              }\n7238524.142023 |   0)   0.080 us    |              paravirt_start_context_switch();\n7238524.142024 |   0)   0.076 us    |              xen_read_cr0();\n7238524.142024 |   0)               |              xen_write_cr0() {\n7238524.142025 |   0)   0.088 us    |                paravirt_get_lazy_mode();\n7238524.142025 |   0)   0.096 us    |                __xen_mc_entry();\n7238524.142026 |   0)   0.083 us    |                paravirt_get_lazy_mode();\n7238524.142026 |   0)   1.802 us    |              }\n7238524.142026 |   0)               |              xen_load_sp0() {\n7238524.142027 |   0)   0.074 us    |                paravirt_get_lazy_mode();\n7238524.142027 |   0)   0.098 us    |                __xen_mc_entry();\n7238524.142028 |   0)   0.073 us    |                paravirt_get_lazy_mode();\n7238524.142029 |   0)   2.289 us    |              }\n7238524.142029 |   0)               |              xen_load_tls() {\n7238524.142029 |   0)   0.073 us    |                paravirt_get_lazy_mode();\n7238524.142030 |   0)   0.079 us    |                paravirt_get_lazy_mode();\n7238524.142031 |   0)   0.135 us    |                load_TLS_descriptor();\n7238524.142031 |   0)   0.082 us    |                load_TLS_descriptor();\n7238524.142032 |   0)   0.091 us    |                load_TLS_descriptor();\n7238524.142032 |   0)   0.081 us    |                paravirt_get_lazy_mode();\n7238524.142033 |   0)   3.306 us    |              }\n7238524.142033 |   0)               |              xen_end_context_switch() {\n7238524.142033 |   0)   0.697 us    |                xen_mc_flush();\n7238524.142034 |   0)   0.083 us    |                paravirt_end_context_switch();\n7238524.142035 |   0)   1.876 us    |              }\n7238524.269404 |   0)               |              finish_task_switch() {\n7238524.269408 |   0)               |                xen_evtchn_do_upcall() {\n7238524.269408 |   0)               |                  irq_enter() {\n7238524.269408 |   0)   0.132 us    |                    rcu_irq_enter();\n7238524.269409 |   0)   0.948 us    |                  }\n7238524.269409 |   0)   0.063 us    |                  exit_idle();\n7238524.269410 |   0)               |                  __xen_evtchn_do_upcall() {\n7238524.269410 |   0)               |                    evtchn_2l_handle_events() {\n7238524.269410 |   0)   0.057 us    |                      irq_from_virq();\n7238524.269411 |   0)               |                      evtchn_from_irq() {\n7238524.269411 |   0)               |                        irq_get_irq_data() {\n7238524.269411 |   0)   0.058 us    |                          irq_to_desc();\n7238524.269412 |   0)   0.579 us    |                        }\n7238524.269412 |   0)   0.898 us    |                      }\n7238524.269412 |   0)               |                      get_evtchn_to_irq() {\n7238524.269412 |   0)   0.049 us    |                        evtchn_2l_max_channels();\n7238524.269412 |   0)   0.390 us    |                      }\n7238524.269413 |   0)               |                      generic_handle_irq() {\n7238524.269413 |   0)   0.051 us    |                        irq_to_desc();\n7238524.269413 |   0)               |                        handle_percpu_irq() {\n7238524.269413 |   0)               |                          ack_dynirq() {\n7238524.269413 |   0)               |                            evtchn_from_irq() {\n7238524.269414 |   0)               |                              irq_get_irq_data() {\n7238524.269414 |   0)   0.057 us    |                                irq_to_desc();\n7238524.269414 |   0)   0.446 us    |                              }\n7238524.269414 |   0)   0.754 us    |                            }\n7238524.269414 |   0)   0.057 us    |                            irq_move_irq();\n7238524.269415 |   0)   0.057 us    |                            evtchn_2l_clear_pending();\n7238524.269415 |   0)   1.718 us    |                          }\n7238524.269415 |   0)               |                          handle_irq_event_percpu() {\n7238524.269416 |   0)               |                            xen_irq_work_interrupt() {\n7238524.269416 |   0)               |                              irq_enter() {\n7238524.269416 |   0)   0.059 us    |                                rcu_irq_enter();\n7238524.269416 |   0)   0.380 us    |                              }\n7238524.269417 |   0)               |                              __wake_up() {\n7238524.269417 |   0)   0.051 us    |                                _raw_spin_lock_irqsave();\n7238524.269417 |   0)               |                                __wake_up_common() {\n7238524.269417 |   0)               |                                  autoremove_wake_function() {\n7238524.269418 |   0)               |                                    default_wake_function() {\n7238524.269418 |   0)               |                                      try_to_wake_up() {\n7238524.269418 |   0)   0.230 us    |                                        _raw_spin_lock_irqsave();\n7238524.269419 |   0)   0.197 us    |                                        task_waking_fair();\n7238524.269419 |   0)               |                                        select_task_rq_fair() {\n7238524.269419 |   0)   0.050 us    |                                          source_load();\n7238524.269420 |   0)   0.057 us    |                                          target_load();\n7238524.269420 |   0)   0.065 us    |                                          idle_cpu();\n7238524.269421 |   0)   0.055 us    |                                          cpus_share_cache();\n7238524.269421 |   0)   0.076 us    |                                          idle_cpu();\n7238524.269421 |   0)   2.041 us    |                                        }\n7238524.269422 |   0)   0.050 us    |                                        _raw_spin_lock();\n7238524.269422 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7238524.269422 |   0)               |                                          activate_task() {\n7238524.269422 |   0)               |                                            enqueue_task() {\n7238524.269422 |   0)   0.175 us    |                                              update_rq_clock();\n7238524.269423 |   0)               |                                              enqueue_task_fair() {\n7238524.269423 |   0)               |                                                enqueue_entity() {\n7238524.269423 |   0)   0.065 us    |                                                  update_curr();\n7238524.269424 |   0)   0.070 us    |                                                  __compute_runnable_contrib.part.51();\n7238524.269424 |   0)   0.052 us    |                                                  __update_entity_load_avg_contrib();\n7238524.269424 |   0)   0.050 us    |                                                  update_cfs_rq_blocked_load();\n7238524.269425 |   0)   0.059 us    |                                                  account_entity_enqueue();\n7238524.269426 |   0)   0.134 us    |                                                  update_cfs_shares();\n7238524.269426 |   0)   0.055 us    |                                                  place_entity();\n7238524.269427 |   0)   0.083 us    |                                                  __enqueue_entity();\n7238524.269427 |   0)   4.026 us    |                                                }\n7238524.269427 |   0)               |                                                enqueue_entity() {\n7238524.269428 |   0)   0.065 us    |                                                  update_curr();\n7238524.269428 |   0)   0.051 us    |                                                  update_cfs_rq_blocked_load();\n7238524.269428 |   0)   0.058 us    |                                                  account_entity_enqueue();\n7238524.269429 |   0)   0.082 us    |                                                  update_cfs_shares();\n7238524.269429 |   0)   0.105 us    |                                                  place_entity();\n7238524.269429 |   0)   0.049 us    |                                                  __enqueue_entity();\n7238524.269430 |   0)   2.247 us    |                                                }\n7238524.269430 |   0)   0.050 us    |                                                hrtick_update();\n7238524.269430 |   0)   7.310 us    |                                              }\n7238524.269430 |   0)   8.101 us    |                                            }\n7238524.269431 |   0)   8.449 us    |                                          }\n7238524.269431 |   0)               |                                          ttwu_do_wakeup() {\n7238524.269431 |   0)               |                                            check_preempt_curr() {\n7238524.269431 |   0)               |                                              resched_task() {\n7238524.269431 |   0)               |                                                xen_smp_send_reschedule() {\n7238524.269432 |   0)               |                                                  xen_send_IPI_one() {\n7238524.269432 |   0)               |                                                    notify_remote_via_irq() {\n7238524.269432 |   0)               |                                                      evtchn_from_irq() {\n7238524.269432 |   0)               |                                                        irq_get_irq_data() {\n7238524.269432 |   0)   0.051 us    |                                                          irq_to_desc();\n7238524.269433 |   0)   0.493 us    |                                                        }\n7238524.269433 |   0)   0.857 us    |                                                      }\n7238524.269434 |   0)   1.909 us    |                                                    } /* notify_remote_via_irq */\n7238524.269434 |   0)   2.288 us    |                                                  }\n7238524.269434 |   0)   2.655 us    |                                                }\n7238524.269434 |   0)   3.127 us    |                                              }\n7238524.269435 |   0)   3.590 us    |                                            }\n7238524.269435 |   0)   4.506 us    |                                          }\n7238524.269436 |   0)   13.594 us   |                                        }\n7238524.269436 |   0)   0.070 us    |                                        _raw_spin_unlock();\n7238524.269436 |   0)   0.163 us    |                                        ttwu_stat();\n7238524.269437 |   0)   0.080 us    |                                        _raw_spin_unlock_irqrestore();\n7238524.269438 |   0)   19.508 us   |                                      }\n7238524.269438 |   0)   19.991 us   |                                    }\n7238524.269438 |   0)   20.486 us   |                                  }\n7238524.269438 |   0)   21.024 us   |                                }\n7238524.269438 |   0)   0.076 us    |                                _raw_spin_unlock_irqrestore();\n7238524.269439 |   0)   22.247 us   |                              }\n7238524.269439 |   0)               |                              irq_exit() {\n7238524.269439 |   0)   0.101 us    |                                idle_cpu();\n7238524.269440 |   0)   0.099 us    |                                rcu_irq_exit();\n7238524.269441 |   0)   1.207 us    |                              }\n7238524.269441 |   0)   25.035 us   |                            }\n7238524.269441 |   0)   0.131 us    |                            add_interrupt_randomness();\n7238524.269442 |   0)   0.076 us    |                            note_interrupt();\n7238524.269442 |   0)   26.909 us   |                          }\n7238524.269443 |   0)   29.377 us   |                        }\n7238524.269443 |   0)   30.139 us   |                      }\n7238524.269443 |   0)   32.759 us   |                    }\n7238524.269443 |   0)   33.204 us   |                  }\n7238524.269444 |   0)               |                  irq_exit() {\n7238524.269444 |   0)               |                    __do_softirq() {\n7238524.269444 |   0)   0.068 us    |                      msecs_to_jiffies();\n7238524.269445 |   0)               |                      rcu_process_callbacks() {\n7238524.269445 |   0)   0.070 us    |                        note_gp_changes();\n7238524.269445 |   0)   0.064 us    |                        _raw_spin_lock_irqsave();\n7238524.269446 |   0)   0.135 us    |                        rcu_accelerate_cbs();\n7238524.269447 |   0)               |                        rcu_report_qs_rnp() {\n7238524.269447 |   0)   0.061 us    |                          _raw_spin_unlock_irqrestore();\n7238524.269448 |   0)   0.779 us    |                        }\n7238524.269448 |   0)   0.081 us    |                        cpu_needs_another_gp();\n7238524.269449 |   0)               |                        file_free_rcu() {\n7238524.269449 |   0)   0.291 us    |                          kmem_cache_free();\n7238524.269450 |   0)   1.139 us    |                        }\n7238524.269451 |   0)               |                        put_cred_rcu() {\n7238524.269451 |   0)               |                          security_cred_free() {\n7238524.269452 |   0)               |                            apparmor_cred_free() {\n7238524.269453 |   0)               |                              aa_free_task_context() {\n7238524.269453 |   0)               |                                kzfree() {\n7238524.269454 |   0)   0.380 us    |                                  ksize();\n7238524.269455 |   0)   0.147 us    |                                  kfree();\n7238524.269455 |   0)   1.602 us    |                                }\n7238524.269455 |   0)   2.631 us    |                              }\n7238524.269456 |   0)   3.611 us    |                            } /* apparmor_cred_free */\n7238524.269456 |   0)   4.927 us    |                          }\n7238524.269457 |   0)   0.071 us    |                          key_put();\n7238524.269457 |   0)   0.071 us    |                          key_put();\n7238524.269458 |   0)   0.065 us    |                          key_put();\n7238524.269458 |   0)   0.066 us    |                          key_put();\n7238524.269459 |   0)   0.390 us    |                          free_uid();\n7238524.269460 |   0)   0.178 us    |                          kmem_cache_free();\n7238524.269460 |   0)   9.429 us    |                        }\n7238524.269461 |   0)   0.099 us    |                        note_gp_changes();\n7238524.269461 |   0)   0.080 us    |                        cpu_needs_another_gp();\n7238524.269462 |   0)   16.796 us   |                      }\n7238524.269462 |   0)   0.068 us    |                      rcu_bh_qs();\n7238524.269462 |   0)   0.066 us    |                      __local_bh_enable();\n7238524.269463 |   0)   18.770 us   |                    }\n7238524.269463 |   0)   0.073 us    |                    idle_cpu();\n7238524.269464 |   0)   0.088 us    |                    rcu_irq_exit();\n7238524.269464 |   0)   20.487 us   |                  }\n7238524.269465 |   0)   56.365 us   |                }\n7238524.269465 |   0)   58.028 us   |              }\n7238524.269466 |   0)   127463.5 us |            }\n7238524.269466 |   0)   127464.2 us |          }\n7238524.269467 |   0)   127465.0 us |        }\n7238524.269467 |   0)   0.095 us    |        down_read();\n7238524.269468 |   0)               |        copy_from_read_buf() {\n7238524.269469 |   0)               |          tty_audit_add_data() {\n7238524.269469 |   0)   0.228 us    |            _raw_spin_lock_irqsave();\n7238524.269470 |   0)   0.070 us    |            _raw_spin_unlock_irqrestore();\n7238524.269471 |   0)   0.074 us    |            _raw_spin_lock_irqsave();\n7238524.269471 |   0)   0.079 us    |            _raw_spin_unlock_irqrestore();\n7238524.269472 |   0)   2.616 us    |          }\n7238524.269472 |   0)   3.878 us    |        }\n7238524.269473 |   0)   0.104 us    |        copy_from_read_buf();\n7238524.269473 |   0)   0.074 us    |        n_tty_set_room();\n7238524.269474 |   0)   0.067 us    |        n_tty_write_wakeup();\n7238524.269474 |   0)               |        __wake_up() {\n7238524.269475 |   0)   0.077 us    |          _raw_spin_lock_irqsave();\n7238524.269475 |   0)               |          __wake_up_common() {\n7238524.269476 |   0)   0.095 us    |            pollwake();\n7238524.269476 |   0)   0.694 us    |          }\n7238524.269476 |   0)   0.064 us    |          _raw_spin_unlock_irqrestore();\n7238524.269477 |   0)   2.128 us    |        }\n7238524.269477 |   0)   0.062 us    |        n_tty_set_room();\n7238524.269477 |   0)   0.066 us    |        up_read();\n7238524.269478 |   0)               |        remove_wait_queue() {\n7238524.269478 |   0)   0.080 us    |          _raw_spin_lock_irqsave();\n7238524.269479 |   0)   0.081 us    |          _raw_spin_unlock_irqrestore();\n7238524.269480 |   0)   1.225 us    |        }\n7238524.269480 |   0)   0.152 us    |        mutex_unlock();\n7238524.269480 |   0)   127485.3 us |      }\n7238524.269481 |   0)               |      tty_ldisc_deref() {\n7238524.269481 |   0)   0.081 us    |        ldsem_up_read();\n7238524.269482 |   0)   0.655 us    |      }\n7238524.269482 |   0)   0.089 us    |      get_seconds();\n7238524.269483 |   0)   127490.1 us |    }\n7238524.269484 |   0)   0.287 us    |    __fsnotify_parent();\n7238524.269484 |   0)   0.183 us    |    fsnotify();\n7238524.269485 |   0)   127496.2 us |  }\n7238524.269559 |   0)               |  vfs_read() {\n7238524.269559 |   0)               |    rw_verify_area() {\n7238524.269560 |   0)               |      security_file_permission() {\n7238524.269560 |   0)               |        apparmor_file_permission() {\n7238524.269561 |   0)   0.164 us    |          common_file_perm();\n7238524.269561 |   0)   0.831 us    |        }\n7238524.269562 |   0)   0.078 us    |        __fsnotify_parent();\n7238524.269562 |   0)   0.080 us    |        fsnotify();\n7238524.269563 |   0)   2.765 us    |      }\n7238524.269563 |   0)   3.490 us    |    }\n7238524.269564 |   0)               |    tty_read() {\n7238524.269564 |   0)   0.066 us    |      tty_paranoia_check();\n7238524.269564 |   0)               |      tty_ldisc_ref_wait() {\n7238524.269565 |   0)   0.085 us    |        ldsem_down_read();\n7238524.269565 |   0)   0.656 us    |      }\n7238524.269566 |   0)               |      n_tty_read() {\n7238524.269566 |   0)   0.078 us    |        _raw_spin_lock_irq();\n7238524.269567 |   0)   0.118 us    |        mutex_lock_interruptible();\n7238524.269567 |   0)   0.078 us    |        down_read();\n7238524.269568 |   0)               |        add_wait_queue() {\n7238524.269568 |   0)   0.089 us    |          _raw_spin_lock_irqsave();\n7238524.269569 |   0)   0.082 us    |          _raw_spin_unlock_irqrestore();\n7238524.269569 |   0)   1.164 us    |        }\n7238524.269570 |   0)   0.073 us    |        tty_hung_up_p();\n7238524.269570 |   0)   0.076 us    |        n_tty_set_room();\n7238524.269571 |   0)   0.078 us    |        up_read();\n7238524.269571 |   0)               |        schedule_timeout() {\n7238524.269572 |   0)               |          schedule() {\n7238524.269572 |   0)               |            __schedule() {\n7238524.269572 |   0)   0.078 us    |              rcu_note_context_switch();\n7238524.269573 |   0)   0.085 us    |              _raw_spin_lock_irq();\n7238524.269574 |   0)               |              deactivate_task() {\n7238524.269574 |   0)               |                dequeue_task() {\n7238524.269574 |   0)   0.185 us    |                  update_rq_clock();\n7238524.269575 |   0)               |                  dequeue_task_fair() {\n7238524.269575 |   0)               |                    dequeue_entity() {\n7238524.269575 |   0)               |                      update_curr() {\n7238524.269576 |   0)   0.206 us    |                        cpuacct_charge();\n7238524.269577 |   0)   0.937 us    |                      }\n7238524.269577 |   0)   0.084 us    |                      __update_entity_load_avg_contrib();\n7238524.269577 |   0)   0.077 us    |                      update_cfs_rq_blocked_load();\n7238524.269578 |   0)   0.075 us    |                      clear_buddies();\n7238524.269579 |   0)   0.096 us    |                      account_entity_dequeue();\n7238524.269579 |   0)               |                      update_cfs_shares() {\n7238524.269580 |   0)   0.095 us    |                        update_curr();\n7238524.269580 |   0)   0.104 us    |                        account_entity_dequeue();\n7238524.269581 |   0)   0.076 us    |                        account_entity_enqueue();\n7238524.269581 |   0)   1.898 us    |                      }\n7238524.269582 |   0)   6.120 us    |                    }\n7238524.269582 |   0)               |                    dequeue_entity() {\n7238524.269582 |   0)   0.093 us    |                      update_curr();\n7238524.269583 |   0)   0.116 us    |                      __update_entity_load_avg_contrib();\n7238524.269583 |   0)   0.085 us    |                      update_cfs_rq_blocked_load();\n7238524.269584 |   0)   0.067 us    |                      clear_buddies();\n7238524.269585 |   0)   0.082 us    |                      account_entity_dequeue();\n7238524.269585 |   0)   0.097 us    |                      update_cfs_shares();\n7238524.269586 |   0)   3.833 us    |                    }\n7238524.269586 |   0)   0.070 us    |                    hrtick_update();\n7238524.269587 |   0)   11.677 us   |                  }\n7238524.269587 |   0)   13.001 us   |                }\n7238524.269587 |   0)   13.516 us   |              }\n7238524.269588 |   0)               |              pick_next_task_fair() {\n7238524.269588 |   0)   0.072 us    |                check_cfs_rq_runtime();\n7238524.269588 |   0)               |                pick_next_entity() {\n7238524.269589 |   0)   0.080 us    |                  clear_buddies();\n7238524.269589 |   0)   0.675 us    |                }\n7238524.269590 |   0)               |                put_prev_entity() {\n7238524.269590 |   0)   0.071 us    |                  check_cfs_rq_runtime();\n7238524.269591 |   0)   0.543 us    |                }\n7238524.269591 |   0)               |                put_prev_entity() {\n7238524.269591 |   0)   0.066 us    |                  check_cfs_rq_runtime();\n7238524.269592 |   0)   0.658 us    |                }\n7238524.269592 |   0)               |                set_next_entity() {\n7238524.269593 |   0)   0.082 us    |                  update_stats_wait_end();\n7238524.269593 |   0)   0.844 us    |                }\n7238524.269594 |   0)   5.970 us    |              }\n7238524.269594 |   0)   0.076 us    |              paravirt_start_context_switch();\n7238524.269595 |   0)   0.074 us    |              xen_read_cr0();\n7238524.269596 |   0)               |              xen_write_cr0() {\n7238524.269597 |   0)   0.081 us    |                paravirt_get_lazy_mode();\n7238524.269597 |   0)   0.086 us    |                __xen_mc_entry();\n7238524.269598 |   0)   0.070 us    |                paravirt_get_lazy_mode();\n7238524.269598 |   0)   1.739 us    |              }\n7238524.269598 |   0)               |              xen_load_sp0() {\n7238524.269599 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7238524.269599 |   0)   0.078 us    |                __xen_mc_entry();\n7238524.269600 |   0)   0.069 us    |                paravirt_get_lazy_mode();\n7238524.269600 |   0)   1.568 us    |              }\n7238524.269601 |   0)               |              xen_load_tls() {\n7238524.269601 |   0)   0.068 us    |                paravirt_get_lazy_mode();\n7238524.269601 |   0)   0.068 us    |                paravirt_get_lazy_mode();\n7238524.269602 |   0)   0.078 us    |                load_TLS_descriptor();\n7238524.269602 |   0)   0.071 us    |                load_TLS_descriptor();\n7238524.269603 |   0)   0.073 us    |                load_TLS_descriptor();\n7238524.269603 |   0)   0.063 us    |                paravirt_get_lazy_mode();\n7238524.269604 |   0)   3.025 us    |              }\n7238524.269604 |   0)               |              xen_end_context_switch() {\n7238524.269604 |   0)   0.646 us    |                xen_mc_flush();\n7238524.269605 |   0)   0.087 us    |                paravirt_end_context_switch();\n7238524.269606 |   0)   1.604 us    |              }\n^C\nEnding tracing...\n\nIf you read through the durations carefully, you can see that the shell begins\nby completing a 19 second read (time between commands), then has a series of\n100 to 200 ms reads (inter-keystroke latency).\n\nThe function times printed are inclusive of their children.\n\n\nThe -C option will print on-CPU times only, excluding sleeping or blocking\nevents from the function duration times. Eg:\n\n# ./funcgraph -Ctp 25285 vfs_read\nTracing \"vfs_read\" for PID 25285... Ctrl-C to end.\n7338520.591816 |   0)               |              finish_task_switch() {\n7338520.591820 |   0)               |                xen_evtchn_do_upcall() {\n7338520.591821 |   0)               |                  irq_enter() {\n7338520.591821 |   0)   0.134 us    |                    rcu_irq_enter();\n7338520.591822 |   0)   0.823 us    |                  }\n7338520.591822 |   0)   0.055 us    |                  exit_idle();\n7338520.591822 |   0)               |                  __xen_evtchn_do_upcall() {\n7338520.591823 |   0)               |                    evtchn_2l_handle_events() {\n7338520.591823 |   0)   0.051 us    |                      irq_from_virq();\n7338520.591823 |   0)               |                      evtchn_from_irq() {\n7338520.591823 |   0)               |                        irq_get_irq_data() {\n7338520.591824 |   0)   0.064 us    |                          irq_to_desc();\n7338520.591824 |   0)   0.572 us    |                        }\n7338520.591824 |   0)   0.973 us    |                      }\n7338520.591825 |   0)               |                      get_evtchn_to_irq() {\n7338520.591825 |   0)   0.049 us    |                        evtchn_2l_max_channels();\n7338520.591825 |   0)   0.386 us    |                      }\n7338520.591825 |   0)               |                      generic_handle_irq() {\n7338520.591825 |   0)   0.061 us    |                        irq_to_desc();\n7338520.591826 |   0)               |                        handle_percpu_irq() {\n7338520.591826 |   0)               |                          ack_dynirq() {\n7338520.591826 |   0)               |                            evtchn_from_irq() {\n7338520.591826 |   0)               |                              irq_get_irq_data() {\n7338520.591827 |   0)   0.050 us    |                                irq_to_desc();\n7338520.591827 |   0)   0.441 us    |                              }\n7338520.591827 |   0)   0.748 us    |                            }\n7338520.591827 |   0)   0.048 us    |                            irq_move_irq();\n7338520.591828 |   0)   0.053 us    |                            evtchn_2l_clear_pending();\n7338520.591828 |   0)   1.810 us    |                          }\n7338520.591828 |   0)               |                          handle_irq_event_percpu() {\n7338520.591828 |   0)               |                            xen_irq_work_interrupt() {\n7338520.591829 |   0)               |                              irq_enter() {\n7338520.591829 |   0)   0.069 us    |                                rcu_irq_enter();\n7338520.591829 |   0)   0.386 us    |                              }\n7338520.591830 |   0)               |                              __wake_up() {\n7338520.591830 |   0)   0.060 us    |                                _raw_spin_lock_irqsave();\n7338520.591830 |   0)               |                                __wake_up_common() {\n7338520.591830 |   0)               |                                  autoremove_wake_function() {\n7338520.591831 |   0)               |                                    default_wake_function() {\n7338520.591831 |   0)               |                                      try_to_wake_up() {\n7338520.591831 |   0)   0.223 us    |                                        _raw_spin_lock_irqsave();\n7338520.591832 |   0)   0.243 us    |                                        task_waking_fair();\n7338520.591832 |   0)               |                                        select_task_rq_fair() {\n7338520.591833 |   0)   0.063 us    |                                          source_load();\n7338520.591833 |   0)   0.059 us    |                                          target_load();\n7338520.591834 |   0)   0.060 us    |                                          idle_cpu();\n7338520.591834 |   0)   0.059 us    |                                          cpus_share_cache();\n7338520.591834 |   0)   0.085 us    |                                          idle_cpu();\n7338520.591835 |   0)   2.176 us    |                                        }\n7338520.591835 |   0)   0.050 us    |                                        _raw_spin_lock();\n7338520.591835 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7338520.591835 |   0)               |                                          activate_task() {\n7338520.591836 |   0)               |                                            enqueue_task() {\n7338520.591836 |   0)   0.197 us    |                                              update_rq_clock();\n7338520.591836 |   0)               |                                              enqueue_task_fair() {\n7338520.591836 |   0)               |                                                enqueue_entity() {\n7338520.591837 |   0)   0.118 us    |                                                  update_curr();\n7338520.591837 |   0)   0.060 us    |                                                  __compute_runnable_contrib.part.51();\n7338520.591838 |   0)   0.052 us    |                                                  __update_entity_load_avg_contrib();\n7338520.591838 |   0)   0.132 us    |                                                  update_cfs_rq_blocked_load();\n7338520.591838 |   0)   0.068 us    |                                                  account_entity_enqueue();\n7338520.591839 |   0)   0.327 us    |                                                  update_cfs_shares();\n7338520.591839 |   0)   0.055 us    |                                                  place_entity();\n7338520.591840 |   0)   0.086 us    |                                                  __enqueue_entity();\n7338520.591840 |   0)   0.069 us    |                                                  update_cfs_rq_blocked_load();\n7338520.591840 |   0)   3.870 us    |                                                }\n7338520.591841 |   0)               |                                                enqueue_entity() {\n7338520.591841 |   0)   0.050 us    |                                                  update_curr();\n7338520.591841 |   0)   0.048 us    |                                                  __compute_runnable_contrib.part.51();\n7338520.591842 |   0)   0.079 us    |                                                  __update_entity_load_avg_contrib();\n7338520.591842 |   0)   0.068 us    |                                                  update_cfs_rq_blocked_load();\n7338520.591842 |   0)   0.072 us    |                                                  account_entity_enqueue();\n7338520.591843 |   0)   0.068 us    |                                                  update_cfs_shares();\n7338520.591844 |   0)   0.123 us    |                                                  place_entity();\n7338520.591844 |   0)   0.051 us    |                                                  __enqueue_entity();\n7338520.591845 |   0)   3.919 us    |                                                }\n7338520.591845 |   0)   0.059 us    |                                                hrtick_update();\n7338520.591845 |   0)   8.895 us    |                                              }\n7338520.591846 |   0)   9.770 us    |                                            }\n7338520.591846 |   0)   10.197 us   |                                          }\n7338520.591846 |   0)               |                                          ttwu_do_wakeup() {\n7338520.591846 |   0)               |                                            check_preempt_curr() {\n7338520.591846 |   0)               |                                              resched_task() {\n7338520.591847 |   0)               |                                                xen_smp_send_reschedule() {\n7338520.591847 |   0)               |                                                  xen_send_IPI_one() {\n7338520.591847 |   0)               |                                                    notify_remote_via_irq() {\n7338520.591847 |   0)               |                                                      evtchn_from_irq() {\n7338520.591848 |   0)               |                                                        irq_get_irq_data() {\n7338520.591848 |   0)   0.051 us    |                                                          irq_to_desc();\n7338520.591848 |   0)   0.503 us    |                                                        }\n7338520.591848 |   0)   1.031 us    |                                                      }\n7338520.591849 |   0)   2.112 us    |                                                    }\n7338520.591849 |   0)   2.484 us    |                                                  }\n7338520.591850 |   0)   2.851 us    |                                                }\n7338520.591850 |   0)   3.311 us    |                                              }\n7338520.591850 |   0)   3.828 us    |                                            }\n7338520.591851 |   0)   4.788 us    |                                          }\n7338520.591851 |   0)   15.731 us   |                                        }\n7338520.591851 |   0)   0.074 us    |                                        _raw_spin_unlock();\n7338520.591852 |   0)   0.156 us    |                                        ttwu_stat();\n7338520.591852 |   0)   0.080 us    |                                        _raw_spin_unlock_irqrestore();\n7338520.591853 |   0)   21.807 us   |                                      }\n7338520.591853 |   0)   22.286 us   |                                    }\n7338520.591853 |   0)   22.738 us   |                                  }\n7338520.591854 |   0)   23.387 us   |                                }\n7338520.591854 |   0)   0.105 us    |                                _raw_spin_unlock_irqrestore();\n7338520.591854 |   0)   24.698 us   |                              }\n7338520.591855 |   0)               |                              irq_exit() {\n7338520.591855 |   0)   0.086 us    |                                idle_cpu();\n7338520.591856 |   0)   0.105 us    |                                rcu_irq_exit();\n7338520.591856 |   0)   1.272 us    |                              }\n7338520.591856 |   0)   27.818 us   |                            }\n7338520.591857 |   0)   0.140 us    |                            add_interrupt_randomness();\n7338520.591857 |   0)   0.084 us    |                            note_interrupt();\n7338520.591858 |   0)   29.866 us   |                          }\n7338520.591858 |   0)   32.390 us   |                        }\n7338520.591859 |   0)   33.204 us   |                      }\n7338520.591859 |   0)   36.137 us   |                    }\n7338520.591859 |   0)   36.574 us   |                  }\n7338520.591859 |   0)               |                  irq_exit() {\n7338520.591860 |   0)   0.073 us    |                    idle_cpu();\n7338520.591860 |   0)   0.076 us    |                    rcu_irq_exit();\n7338520.591861 |   0)   1.091 us    |                  }\n7338520.591861 |   0)   40.156 us   |                }\n7338520.591862 |   0)   41.874 us   |              }\n7338520.591862 |   0)   75.633 us   |            } /* __schedule */\n7338520.591862 |   0)   76.182 us   |          } /* schedule */\n7338520.591863 |   0)   76.965 us   |        } /* schedule_timeout */\n7338520.591863 |   0)   0.070 us    |        down_read();\n7338520.591864 |   0)               |        copy_from_read_buf() {\n7338520.591865 |   0)               |          tty_audit_add_data() {\n7338520.591865 |   0)   0.232 us    |            _raw_spin_lock_irqsave();\n7338520.591866 |   0)   0.079 us    |            _raw_spin_unlock_irqrestore();\n7338520.591867 |   0)   0.122 us    |            _raw_spin_lock_irqsave();\n7338520.591867 |   0)   0.066 us    |            _raw_spin_unlock_irqrestore();\n7338520.591868 |   0)   2.642 us    |          }\n7338520.591868 |   0)   3.886 us    |        }\n7338520.591868 |   0)   0.149 us    |        copy_from_read_buf();\n7338520.591869 |   0)   0.072 us    |        n_tty_set_room();\n7338520.591870 |   0)   0.071 us    |        n_tty_write_wakeup();\n7338520.591870 |   0)               |        __wake_up() {\n7338520.591871 |   0)   0.071 us    |          _raw_spin_lock_irqsave();\n7338520.591872 |   0)               |          __wake_up_common() {\n7338520.591872 |   0)   0.097 us    |            pollwake();\n7338520.591873 |   0)   0.739 us    |          }\n7338520.591873 |   0)   0.066 us    |          _raw_spin_unlock_irqrestore();\n7338520.591874 |   0)   3.043 us    |        }\n7338520.591874 |   0)   0.075 us    |        n_tty_set_room();\n7338520.591875 |   0)   0.106 us    |        up_read();\n7338520.591875 |   0)               |        remove_wait_queue() {\n7338520.591875 |   0)   0.078 us    |          _raw_spin_lock_irqsave();\n7338520.591876 |   0)   0.075 us    |          _raw_spin_unlock_irqrestore();\n7338520.591877 |   0)   1.165 us    |        }\n7338520.591877 |   0)   0.137 us    |        mutex_unlock();\n7338520.591877 |   0)   98.321 us   |      } /* n_tty_read */\n7338520.591878 |   0)               |      tty_ldisc_deref() {\n7338520.591878 |   0)   0.072 us    |        ldsem_up_read();\n7338520.591879 |   0)   0.561 us    |      }\n7338520.591879 |   0)   0.090 us    |      get_seconds();\n7338520.591880 |   0)   102.599 us  |    } /* tty_read */\n7338520.591880 |   0)   0.362 us    |    __fsnotify_parent();\n7338520.591881 |   0)   0.171 us    |    fsnotify();\n7338520.591882 |   0)   109.640 us  |  } /* vfs_read */\n7338520.591951 |   0)               |  vfs_read() {\n7338520.591951 |   0)               |    rw_verify_area() {\n7338520.591952 |   0)               |      security_file_permission() {\n7338520.591952 |   0)               |        apparmor_file_permission() {\n7338520.591952 |   0)   0.174 us    |          common_file_perm();\n7338520.591953 |   0)   0.762 us    |        }\n7338520.591953 |   0)   0.126 us    |        __fsnotify_parent();\n7338520.591954 |   0)   0.088 us    |        fsnotify();\n7338520.591954 |   0)   2.609 us    |      }\n7338520.591955 |   0)   3.351 us    |    }\n7338520.591955 |   0)               |    tty_read() {\n7338520.591956 |   0)   0.081 us    |      tty_paranoia_check();\n7338520.591956 |   0)               |      tty_ldisc_ref_wait() {\n7338520.591956 |   0)   0.090 us    |        ldsem_down_read();\n7338520.591957 |   0)   0.633 us    |      }\n7338520.591957 |   0)               |      n_tty_read() {\n7338520.591958 |   0)   0.073 us    |        _raw_spin_lock_irq();\n7338520.591958 |   0)   0.089 us    |        mutex_lock_interruptible();\n7338520.591959 |   0)   0.080 us    |        down_read();\n7338520.591960 |   0)               |        add_wait_queue() {\n7338520.591960 |   0)   0.084 us    |          _raw_spin_lock_irqsave();\n7338520.591960 |   0)   0.087 us    |          _raw_spin_unlock_irqrestore();\n7338520.591961 |   0)   1.215 us    |        }\n7338520.591961 |   0)   0.078 us    |        tty_hung_up_p();\n7338520.591962 |   0)   0.084 us    |        n_tty_set_room();\n7338520.591962 |   0)   0.072 us    |        up_read();\n7338520.591963 |   0)               |        schedule_timeout() {\n7338520.591963 |   0)               |          schedule() {\n7338520.591964 |   0)               |            __schedule() {\n7338520.591964 |   0)   0.084 us    |              rcu_note_context_switch();\n7338520.591965 |   0)   0.086 us    |              _raw_spin_lock_irq();\n7338520.591965 |   0)               |              deactivate_task() {\n7338520.591966 |   0)               |                dequeue_task() {\n7338520.591966 |   0)   0.171 us    |                  update_rq_clock();\n7338520.591966 |   0)               |                  dequeue_task_fair() {\n7338520.591967 |   0)               |                    dequeue_entity() {\n7338520.591967 |   0)               |                      update_curr() {\n7338520.591967 |   0)   0.248 us    |                        cpuacct_charge();\n7338520.591968 |   0)   0.974 us    |                      }\n7338520.591969 |   0)   0.074 us    |                      update_cfs_rq_blocked_load();\n7338520.591969 |   0)   0.081 us    |                      clear_buddies();\n7338520.591970 |   0)   0.094 us    |                      account_entity_dequeue();\n7338520.591971 |   0)               |                      update_cfs_shares() {\n7338520.591971 |   0)   0.096 us    |                        update_curr();\n7338520.591971 |   0)   0.093 us    |                        account_entity_dequeue();\n7338520.591972 |   0)   0.079 us    |                        account_entity_enqueue();\n7338520.591972 |   0)   1.743 us    |                      }\n7338520.591972 |   0)   5.515 us    |                    }\n7338520.591973 |   0)               |                    dequeue_entity() {\n7338520.591973 |   0)   0.088 us    |                      update_curr();\n7338520.591974 |   0)   0.106 us    |                      update_cfs_rq_blocked_load();\n7338520.591975 |   0)   0.078 us    |                      clear_buddies();\n7338520.591975 |   0)   0.088 us    |                      account_entity_dequeue();\n7338520.591976 |   0)   0.091 us    |                      update_cfs_shares();\n7338520.591977 |   0)   3.639 us    |                    }\n7338520.591977 |   0)   0.078 us    |                    hrtick_update();\n7338520.591978 |   0)   10.851 us   |                  }\n7338520.591978 |   0)   11.992 us   |                }\n7338520.591978 |   0)   12.496 us   |              }\n7338520.591978 |   0)               |              pick_next_task_fair() {\n7338520.591979 |   0)   0.079 us    |                check_cfs_rq_runtime();\n7338520.591979 |   0)               |                pick_next_entity() {\n7338520.591979 |   0)   0.080 us    |                  clear_buddies();\n7338520.591980 |   0)   0.594 us    |                }\n7338520.591980 |   0)               |                put_prev_entity() {\n7338520.591980 |   0)   0.078 us    |                  check_cfs_rq_runtime();\n7338520.591981 |   0)   0.641 us    |                }\n7338520.591981 |   0)               |                put_prev_entity() {\n7338520.591982 |   0)   0.076 us    |                  check_cfs_rq_runtime();\n7338520.591982 |   0)   0.610 us    |                }\n7338520.591982 |   0)               |                set_next_entity() {\n7338520.591983 |   0)   0.097 us    |                  update_stats_wait_end();\n7338520.591983 |   0)   0.744 us    |                }\n7338520.591984 |   0)   5.115 us    |              }\n7338520.591984 |   0)   0.076 us    |              paravirt_start_context_switch();\n7338520.591985 |   0)   0.086 us    |              xen_read_cr0();\n7338520.591986 |   0)               |              xen_write_cr0() {\n7338520.591986 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7338520.591987 |   0)   0.086 us    |                __xen_mc_entry();\n7338520.591987 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7338520.591988 |   0)   1.698 us    |              }\n7338520.591988 |   0)               |              xen_load_sp0() {\n7338520.591988 |   0)   0.074 us    |                paravirt_get_lazy_mode();\n7338520.591989 |   0)   0.084 us    |                __xen_mc_entry();\n7338520.591989 |   0)   0.084 us    |                paravirt_get_lazy_mode();\n7338520.591990 |   0)   1.724 us    |              }\n7338520.591990 |   0)               |              xen_load_tls() {\n7338520.591991 |   0)   0.080 us    |                paravirt_get_lazy_mode();\n7338520.591991 |   0)   0.088 us    |                paravirt_get_lazy_mode();\n7338520.591992 |   0)   0.140 us    |                load_TLS_descriptor();\n7338520.591992 |   0)   0.079 us    |                load_TLS_descriptor();\n7338520.591993 |   0)   0.087 us    |                load_TLS_descriptor();\n7338520.591994 |   0)   0.078 us    |                paravirt_get_lazy_mode();\n7338520.591994 |   0)   3.666 us    |              }\n7338520.591995 |   0)               |              xen_end_context_switch() {\n7338520.591995 |   0)   0.644 us    |                xen_mc_flush();\n7338520.591996 |   0)   0.080 us    |                paravirt_end_context_switch();\n7338520.591997 |   0)   1.813 us    |              }\n7338520.855105 |   0)               |              finish_task_switch() {\n7338520.855110 |   0)               |                xen_evtchn_do_upcall() {\n7338520.855110 |   0)               |                  irq_enter() {\n7338520.855110 |   0)   0.137 us    |                    rcu_irq_enter();\n7338520.855111 |   0)   0.673 us    |                  }\n7338520.855111 |   0)   0.063 us    |                  exit_idle();\n7338520.855111 |   0)               |                  __xen_evtchn_do_upcall() {\n7338520.855112 |   0)               |                    evtchn_2l_handle_events() {\n7338520.855112 |   0)   0.050 us    |                      irq_from_virq();\n7338520.855112 |   0)               |                      evtchn_from_irq() {\n7338520.855112 |   0)               |                        irq_get_irq_data() {\n7338520.855113 |   0)   0.050 us    |                          irq_to_desc();\n7338520.855113 |   0)   0.568 us    |                        }\n7338520.855113 |   0)   0.895 us    |                      }\n7338520.855114 |   0)               |                      get_evtchn_to_irq() {\n7338520.855114 |   0)   0.048 us    |                        evtchn_2l_max_channels();\n7338520.855114 |   0)   0.386 us    |                      }\n7338520.855114 |   0)               |                      generic_handle_irq() {\n7338520.855114 |   0)   0.051 us    |                        irq_to_desc();\n7338520.855115 |   0)               |                        handle_percpu_irq() {\n7338520.855115 |   0)               |                          ack_dynirq() {\n7338520.855115 |   0)               |                            evtchn_from_irq() {\n7338520.855115 |   0)               |                              irq_get_irq_data() {\n7338520.855116 |   0)   0.058 us    |                                irq_to_desc();\n7338520.855117 |   0)   1.264 us    |                              }\n7338520.855117 |   0)   1.644 us    |                            }\n7338520.855117 |   0)   0.048 us    |                            irq_move_irq();\n7338520.855118 |   0)   0.050 us    |                            evtchn_2l_clear_pending();\n7338520.855118 |   0)   2.876 us    |                          }\n7338520.855118 |   0)               |                          handle_irq_event_percpu() {\n7338520.855119 |   0)               |                            xen_irq_work_interrupt() {\n7338520.855119 |   0)               |                              irq_enter() {\n7338520.855119 |   0)   0.055 us    |                                rcu_irq_enter();\n7338520.855119 |   0)   0.460 us    |                              }\n7338520.855120 |   0)               |                              __wake_up() {\n7338520.855120 |   0)   0.057 us    |                                _raw_spin_lock_irqsave();\n7338520.855120 |   0)               |                                __wake_up_common() {\n7338520.855121 |   0)               |                                  autoremove_wake_function() {\n7338520.855121 |   0)               |                                    default_wake_function() {\n7338520.855121 |   0)               |                                      try_to_wake_up() {\n7338520.855121 |   0)   0.203 us    |                                        _raw_spin_lock_irqsave();\n7338520.855122 |   0)   0.179 us    |                                        task_waking_fair();\n7338520.855123 |   0)               |                                        select_task_rq_fair() {\n7338520.855123 |   0)   0.048 us    |                                          source_load();\n7338520.855123 |   0)   0.059 us    |                                          target_load();\n7338520.855124 |   0)   0.059 us    |                                          idle_cpu();\n7338520.855124 |   0)   0.058 us    |                                          cpus_share_cache();\n7338520.855124 |   0)   0.058 us    |                                          idle_cpu();\n7338520.855125 |   0)   1.940 us    |                                        }\n7338520.855125 |   0)   0.057 us    |                                        _raw_spin_lock();\n7338520.855125 |   0)               |                                        ttwu_do_activate.constprop.124() {\n7338520.855125 |   0)               |                                          activate_task() {\n7338520.855126 |   0)               |                                            enqueue_task() {\n7338520.855126 |   0)   0.171 us    |                                              update_rq_clock();\n7338520.855126 |   0)               |                                              enqueue_task_fair() {\n7338520.855126 |   0)               |                                                enqueue_entity() {\n7338520.855127 |   0)   0.063 us    |                                                  update_curr();\n7338520.855127 |   0)   0.078 us    |                                                  __compute_runnable_contrib.part.51();\n7338520.855127 |   0)   0.066 us    |                                                  __update_entity_load_avg_contrib();\n7338520.855128 |   0)   0.061 us    |                                                  update_cfs_rq_blocked_load();\n7338520.855128 |   0)   0.072 us    |                                                  account_entity_enqueue();\n7338520.855128 |   0)   0.116 us    |                                                  update_cfs_shares();\n7338520.855129 |   0)   0.062 us    |                                                  place_entity();\n7338520.855129 |   0)   0.087 us    |                                                  __enqueue_entity();\n7338520.855129 |   0)   2.950 us    |                                                }\n7338520.855130 |   0)               |                                                enqueue_entity() {\n7338520.855130 |   0)   0.065 us    |                                                  update_curr();\n7338520.855130 |   0)   0.065 us    |                                                  update_cfs_rq_blocked_load();\n7338520.855130 |   0)   0.067 us    |                                                  account_entity_enqueue();\n7338520.855131 |   0)   0.084 us    |                                                  update_cfs_shares();\n7338520.855131 |   0)   0.112 us    |                                                  place_entity();\n7338520.855131 |   0)   0.051 us    |                                                  __enqueue_entity();\n7338520.855132 |   0)   2.074 us    |                                                }\n7338520.855132 |   0)   0.055 us    |                                                hrtick_update();\n7338520.855132 |   0)   5.983 us    |                                              }\n7338520.855133 |   0)   6.790 us    |                                            }\n7338520.855133 |   0)   7.138 us    |                                          }\n7338520.855133 |   0)               |                                          ttwu_do_wakeup() {\n7338520.855133 |   0)               |                                            check_preempt_curr() {\n7338520.855133 |   0)               |                                              resched_task() {\n7338520.855133 |   0)               |                                                xen_smp_send_reschedule() {\n7338520.855134 |   0)               |                                                  xen_send_IPI_one() {\n7338520.855134 |   0)               |                                                    notify_remote_via_irq() {\n7338520.855134 |   0)               |                                                      evtchn_from_irq() {\n7338520.855134 |   0)               |                                                        irq_get_irq_data() {\n7338520.855134 |   0)   0.057 us    |                                                          irq_to_desc();\n7338520.855135 |   0)   0.502 us    |                                                        }\n7338520.855135 |   0)   0.865 us    |                                                      }\n7338520.855136 |   0)   1.975 us    |                                                    } /* notify_remote_via_irq */\n7338520.855136 |   0)   2.350 us    |                                                  }\n7338520.855136 |   0)   2.723 us    |                                                }\n7338520.855136 |   0)   3.175 us    |                                              }\n7338520.855137 |   0)   3.620 us    |                                            }\n7338520.855138 |   0)   4.642 us    |                                          }\n7338520.855138 |   0)   12.409 us   |                                        }\n7338520.855138 |   0)   0.059 us    |                                        _raw_spin_unlock();\n7338520.855139 |   0)   0.108 us    |                                        ttwu_stat();\n7338520.855140 |   0)   0.073 us    |                                        _raw_spin_unlock_irqrestore();\n7338520.855140 |   0)   18.857 us   |                                      }\n7338520.855141 |   0)   19.415 us   |                                    }\n7338520.855141 |   0)   19.993 us   |                                  }\n7338520.855141 |   0)   20.587 us   |                                }\n7338520.855141 |   0)   0.070 us    |                                _raw_spin_unlock_irqrestore();\n7338520.855142 |   0)   21.858 us   |                              }\n7338520.855142 |   0)               |                              irq_exit() {\n7338520.855143 |   0)   0.084 us    |                                idle_cpu();\n7338520.855143 |   0)   0.082 us    |                                rcu_irq_exit();\n7338520.855144 |   0)   1.235 us    |                              }\n7338520.855144 |   0)   25.109 us   |                            }\n7338520.855144 |   0)   0.126 us    |                            add_interrupt_randomness();\n7338520.855145 |   0)   0.091 us    |                            note_interrupt();\n7338520.855145 |   0)   26.935 us   |                          }\n7338520.855146 |   0)   30.693 us   |                        }\n7338520.855146 |   0)   31.575 us   |                      }\n7338520.855146 |   0)   34.424 us   |                    }\n7338520.855147 |   0)   34.841 us   |                  }\n7338520.855147 |   0)               |                  irq_exit() {\n7338520.855147 |   0)   0.083 us    |                    idle_cpu();\n7338520.855148 |   0)   0.069 us    |                    rcu_irq_exit();\n7338520.855148 |   0)   1.056 us    |                  }\n7338520.855148 |   0)   38.284 us   |                }\n7338520.855149 |   0)   39.892 us   |              }\n7338520.855150 |   0)   72.181 us   |            }\n7338520.855150 |   0)   72.925 us   |          }\n7338520.855150 |   0)   73.638 us   |        }\n7338520.855151 |   0)   0.078 us    |        down_read();\n7338520.855152 |   0)               |        copy_from_read_buf() {\n7338520.855153 |   0)               |          tty_audit_add_data() {\n7338520.855153 |   0)   0.272 us    |            _raw_spin_lock_irqsave();\n7338520.855154 |   0)   0.063 us    |            _raw_spin_unlock_irqrestore();\n7338520.855155 |   0)   0.086 us    |            _raw_spin_lock_irqsave();\n7338520.855155 |   0)   0.067 us    |            _raw_spin_unlock_irqrestore();\n7338520.855156 |   0)   2.808 us    |          }\n7338520.855156 |   0)   4.330 us    |        }\n7338520.855156 |   0)   0.083 us    |        copy_from_read_buf();\n7338520.855157 |   0)   0.062 us    |        n_tty_set_room();\n7338520.855158 |   0)   0.079 us    |        n_tty_write_wakeup();\n7338520.855158 |   0)               |        __wake_up() {\n7338520.855158 |   0)   0.068 us    |          _raw_spin_lock_irqsave();\n7338520.855159 |   0)               |          __wake_up_common() {\n7338520.855159 |   0)   0.092 us    |            pollwake();\n7338520.855160 |   0)   0.643 us    |          }\n7338520.855160 |   0)   0.074 us    |          _raw_spin_unlock_irqrestore();\n7338520.855160 |   0)   2.040 us    |        }\n7338520.855161 |   0)   0.074 us    |        n_tty_set_room();\n7338520.855162 |   0)   0.073 us    |        up_read();\n7338520.855162 |   0)               |        remove_wait_queue() {\n7338520.855162 |   0)   0.084 us    |          _raw_spin_lock_irqsave();\n7338520.855163 |   0)   0.078 us    |          _raw_spin_unlock_irqrestore();\n7338520.855163 |   0)   1.166 us    |        }\n7338520.855164 |   0)   0.140 us    |        mutex_unlock();\n7338520.855164 |   0)   93.360 us   |      }\n7338520.855165 |   0)               |      tty_ldisc_deref() {\n7338520.855165 |   0)   0.070 us    |        ldsem_up_read();\n7338520.855166 |   0)   0.746 us    |      }\n7338520.855166 |   0)   0.071 us    |      get_seconds();\n7338520.855167 |   0)   97.713 us   |    }\n7338520.855167 |   0)   0.283 us    |    __fsnotify_parent();\n7338520.855168 |   0)   0.172 us    |    fsnotify();\n7338520.855168 |   0)   103.847 us  |  }\n7338520.855238 |   0)               |  vfs_read() {\n7338520.855239 |   0)               |    rw_verify_area() {\n7338520.855240 |   0)               |      security_file_permission() {\n7338520.855240 |   0)               |        apparmor_file_permission() {\n7338520.855240 |   0)   0.160 us    |          common_file_perm();\n7338520.855241 |   0)   0.770 us    |        }\n7338520.855241 |   0)   0.078 us    |        __fsnotify_parent();\n7338520.855242 |   0)   0.087 us    |        fsnotify();\n7338520.855243 |   0)   2.595 us    |      }\n7338520.855243 |   0)   4.148 us    |    }\n7338520.855243 |   0)               |    tty_read() {\n7338520.855244 |   0)   0.078 us    |      tty_paranoia_check();\n7338520.855244 |   0)               |      tty_ldisc_ref_wait() {\n7338520.855244 |   0)   0.084 us    |        ldsem_down_read();\n7338520.855245 |   0)   0.643 us    |      }\n7338520.855245 |   0)               |      n_tty_read() {\n7338520.855246 |   0)   0.079 us    |        _raw_spin_lock_irq();\n7338520.855247 |   0)   0.171 us    |        mutex_lock_interruptible();\n7338520.855247 |   0)   0.064 us    |        down_read();\n7338520.855248 |   0)               |        add_wait_queue() {\n7338520.855248 |   0)   0.078 us    |          _raw_spin_lock_irqsave();\n7338520.855249 |   0)   0.082 us    |          _raw_spin_unlock_irqrestore();\n7338520.855249 |   0)   1.076 us    |        }\n7338520.855250 |   0)   0.075 us    |        tty_hung_up_p();\n7338520.855250 |   0)   0.079 us    |        n_tty_set_room();\n7338520.855251 |   0)   0.075 us    |        up_read();\n7338520.855251 |   0)               |        schedule_timeout() {\n7338520.855252 |   0)               |          schedule() {\n7338520.855252 |   0)               |            __schedule() {\n7338520.855252 |   0)   0.084 us    |              rcu_note_context_switch();\n7338520.855253 |   0)   0.079 us    |              _raw_spin_lock_irq();\n7338520.855254 |   0)               |              deactivate_task() {\n7338520.855254 |   0)               |                dequeue_task() {\n7338520.855254 |   0)   0.219 us    |                  update_rq_clock();\n7338520.855255 |   0)               |                  dequeue_task_fair() {\n7338520.855255 |   0)               |                    dequeue_entity() {\n7338520.855255 |   0)               |                      update_curr() {\n7338520.855256 |   0)   0.186 us    |                        cpuacct_charge();\n7338520.855257 |   0)   0.924 us    |                      }\n7338520.855257 |   0)   0.078 us    |                      update_cfs_rq_blocked_load();\n7338520.855258 |   0)   0.078 us    |                      clear_buddies();\n7338520.855258 |   0)   0.083 us    |                      account_entity_dequeue();\n7338520.855259 |   0)               |                      update_cfs_shares() {\n7338520.855259 |   0)   0.105 us    |                        update_curr();\n7338520.855260 |   0)   0.093 us    |                        account_entity_dequeue();\n7338520.855260 |   0)   0.098 us    |                        account_entity_enqueue();\n7338520.855261 |   0)   1.825 us    |                      }\n7338520.855261 |   0)   5.574 us    |                    }\n7338520.855261 |   0)               |                    dequeue_entity() {\n7338520.855261 |   0)   0.086 us    |                      update_curr();\n7338520.855262 |   0)   0.127 us    |                      __update_entity_load_avg_contrib();\n7338520.855263 |   0)   0.070 us    |                      update_cfs_rq_blocked_load();\n7338520.855263 |   0)   0.066 us    |                      clear_buddies();\n7338520.855264 |   0)   0.082 us    |                      account_entity_dequeue();\n7338520.855264 |   0)   0.104 us    |                      update_cfs_shares();\n7338520.855265 |   0)   3.439 us    |                    }\n7338520.855265 |   0)   0.078 us    |                    hrtick_update();\n7338520.855266 |   0)   10.741 us   |                  }\n7338520.855266 |   0)   11.990 us   |                }\n7338520.855266 |   0)   12.580 us   |              }\n7338520.855267 |   0)               |              pick_next_task_fair() {\n7338520.855267 |   0)   0.074 us    |                check_cfs_rq_runtime();\n7338520.855268 |   0)               |                pick_next_entity() {\n7338520.855268 |   0)   0.078 us    |                  clear_buddies();\n7338520.855269 |   0)   0.696 us    |                }\n7338520.855269 |   0)               |                put_prev_entity() {\n7338520.855269 |   0)   0.084 us    |                  check_cfs_rq_runtime();\n7338520.855270 |   0)   0.628 us    |                }\n7338520.855270 |   0)               |                put_prev_entity() {\n7338520.855270 |   0)   0.074 us    |                  check_cfs_rq_runtime();\n7338520.855271 |   0)   0.575 us    |                }\n7338520.855271 |   0)               |                set_next_entity() {\n7338520.855272 |   0)   0.104 us    |                  update_stats_wait_end();\n7338520.855273 |   0)   0.834 us    |                }\n7338520.855273 |   0)   5.872 us    |              }\n7338520.855274 |   0)   0.079 us    |              paravirt_start_context_switch();\n7338520.855275 |   0)   0.080 us    |              xen_read_cr0();\n7338520.855276 |   0)               |              xen_write_cr0() {\n7338520.855276 |   0)   0.091 us    |                paravirt_get_lazy_mode();\n7338520.855277 |   0)   0.087 us    |                __xen_mc_entry();\n7338520.855277 |   0)   0.076 us    |                paravirt_get_lazy_mode();\n7338520.855278 |   0)   1.986 us    |              }\n7338520.855278 |   0)               |              xen_load_sp0() {\n7338520.855278 |   0)   0.066 us    |                paravirt_get_lazy_mode();\n7338520.855279 |   0)   0.083 us    |                __xen_mc_entry();\n7338520.855280 |   0)   0.082 us    |                paravirt_get_lazy_mode();\n7338520.855280 |   0)   1.925 us    |              }\n7338520.855281 |   0)               |              xen_load_tls() {\n7338520.855281 |   0)   0.082 us    |                paravirt_get_lazy_mode();\n7338520.855281 |   0)   0.080 us    |                paravirt_get_lazy_mode();\n7338520.855282 |   0)   0.137 us    |                load_TLS_descriptor();\n7338520.855283 |   0)   0.090 us    |                load_TLS_descriptor();\n7338520.855283 |   0)   0.081 us    |                load_TLS_descriptor();\n7338520.855284 |   0)   0.081 us    |                paravirt_get_lazy_mode();\n7338520.855284 |   0)   3.397 us    |              }\n7338520.855284 |   0)               |              xen_end_context_switch() {\n7338520.855285 |   0)   0.618 us    |                xen_mc_flush();\n7338520.855286 |   0)   0.086 us    |                paravirt_end_context_switch();\n7338520.855286 |   0)   1.708 us    |              }\n^C\nEnding tracing...\n\nUnderstanding whether the time is on-CPU or blocked off-CPU directs the\nperformance investigation.\n\n\nUse -h to print the USAGE message:\n\n# ./funcgraph -h\nUSAGE: funcgraph [-aCDhHPtT] [-m maxdepth] [-p PID] [-L TID] [-d secs] funcstring\n                 -a              # all info (same as -HPt)\n                 -C              # measure on-CPU time only\n                 -d seconds      # trace duration, and use buffers\n                 -D              # do not show function duration\n                 -h              # this usage message\n                 -H              # include column headers\n                 -m maxdepth     # max stack depth to show\n                 -p PID          # trace when this pid is on-CPU\n                 -L TID          # trace when this thread is on-CPU\n                 -P              # show process names & PIDs\n                 -t              # show timestamps\n                 -T              # comment function tails\n  eg,\n       funcgraph do_nanosleep    # trace do_nanosleep() and children\n       funcgraph -m 3 do_sys_open # trace do_sys_open() to 3 levels only\n       funcgraph -a do_sys_open    # include timestamps and process name\n       funcgraph -p 198 do_sys_open # trace vfs_read() for PID 198 only\n       funcgraph -d 1 do_sys_open >out # trace 1 sec, then write to file\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/funcslower_example.txt",
    "content": "Demonstrations of funcslower, the Linux ftrace version.\n\n\nShow me ext3_readpages() calls slower than 1000 microseconds (1 ms):\n\n# ./funcslower ext3_readpages 1000\nTracing \"ext3_readpages\" slower than 1000 us... Ctrl-C to end.\n 0) ! 8147.120 us |  } /* ext3_readpages */\n 0) ! 8135.067 us |  } /* ext3_readpages */\n 0) ! 12202.93 us |  } /* ext3_readpages */\n 0) ! 12201.84 us |  } /* ext3_readpages */\n 0) ! 8142.667 us |  } /* ext3_readpages */\n 0) ! 12194.14 us |  } /* ext3_readpages */\n^C\nEnding tracing...\n\nNeat. So this confirms that there are ext3_readpages() calls that are taking\nover 8000 us (8 ms).\n\nfuncslower uses the ftrace function graph profiler to dynamically instrument\nthe given kernel function, time it in-kernel, and only emit events slower\nthan the given latency threshold in-kernel. Since this all operates in\nkernel context, the overheads are relatively low (compared to post-processing\nin user space).\n\n\nNow include the process name and PID (-P) of the process who is on-CPU, and the\nabsolute timestamp (-t) of the event:\n\n# ./funcslower -Pt ext3_readpages 1000\nTracing \"ext3_readpages\" slower than 1000 us... Ctrl-C to end.\n2678112.003180 |   0)  cksum-26695   | ! 8145.268 us |  } /* ext3_readpages */\n2678113.538763 |   0)  cksum-26695   | ! 8139.086 us |  } /* ext3_readpages */\n2678113.704901 |   0)  cksum-26695   | ! 8147.549 us |  } /* ext3_readpages */\n2678113.721102 |   0)  cksum-26695   | ! 8142.530 us |  } /* ext3_readpages */\n2678113.810269 |   0)  cksum-26695   | ! 12234.70 us |  } /* ext3_readpages */\n2678113.996625 |   0)  cksum-26695   | ! 8146.129 us |  } /* ext3_readpages */\n2678114.012832 |   0)  cksum-26695   | ! 8148.153 us |  } /* ext3_readpages */\n^C\nEnding tracing...\n\nGreat! Now I can see the process name, which in this case is the responsible\nprocess. The timestamps also let me determine the rate of these slow events.\n\n\nNow measure time differently: excluding time spent sleeping, so that we only\nsee on-CPU time:\n\n# ./funcslower -Pct ext3_readpages 1000\nTracing \"ext3_readpages\" slower than 1000 us... Ctrl-C to end.\n^C\nEnding tracing...\n\nI believe the workload hasn't changed, so these ext3_readpages() calls are\nstill happening, however, their CPU time doesn't exceed 1 ms. Compared to the\nearlier output, this tells me that the latency in this function is due to time\nspent blocked off-CPU, and not on-CPU. This makes sense: this function is\nultimately being blocked on disk I/O.\n\nWere the function duration times to be similar with and without -C, that would\ntell us that the high latency is due to time spent on-CPU executing code.\n\n\nThis traces the sys_nanosleep() kernel function, and shows calls taking over\n100 us:\n\n# ./funcslower sys_nanosleep 100\nTracing \"sys_nanosleep\" slower than 100 us... Ctrl-C to end.\n 0) ! 2000147 us |  } /* sys_nanosleep */\n ------------------------------------------\n 0) registe-27414  =>  vmstat-27419 \n ------------------------------------------\n\n 0) ! 1000143 us |  } /* sys_nanosleep */\n 0) ! 1000154 us |  } /* sys_nanosleep */\n ------------------------------------------\n 0)  vmstat-27419  => registe-27414 \n ------------------------------------------\n\n 0) ! 2000183 us |  } /* sys_nanosleep */\n ------------------------------------------\n 0) registe-27414  =>  vmstat-27419 \n ------------------------------------------\n\n 0) ! 1000141 us |  } /* sys_nanosleep */\n^C\nEnding tracing...\n\nThis is an example where I did not use -P, but ftrace has included process\ninformation anyway. Look for the lines containing \"=>\", which indicate a process\nswitch on the given CPU.\n\n\nUse -h to print the USAGE message:\n\n# ./funcslower -h\nUSAGE: funcslower [-aChHPt] [-p PID] [-d secs] funcstring latency_us\n                 -a              # all info (same as -HPt)\n                 -C              # measure on-CPU time only\n                 -d seconds      # trace duration, and use buffers\n                 -h              # this usage message\n                 -H              # include column headers\n                 -p PID          # trace when this pid is on-CPU\n                 -L TID          # trace when this thread is on-CPU\n                 -P              # show process names & PIDs\n                 -t              # show timestamps\n  eg,\n       funcslower vfs_read 10000 # trace vfs_read() slower than 10 ms\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/functrace_example.txt",
    "content": "Demonstrations of functrace, the Linux ftrace version.\n\n\nA (usually) good example to start with is do_nanosleep(), since it is not called\nfrequently, and easily triggered. Here's tracing it using functrace:\n\n# ./functrace 'do_nanosleep'\nTracing \"do_nanosleep\"... Ctrl-C to end.\n          svscan-1678  [000] .... 6412438.703521: do_nanosleep <-hrtimer_nanosleep\n          svscan-1678  [000] .... 6412443.703678: do_nanosleep <-hrtimer_nanosleep\n          svscan-1678  [000] .... 6412448.703865: do_nanosleep <-hrtimer_nanosleep\n          vmstat-28371 [000] .... 6412453.216241: do_nanosleep <-hrtimer_nanosleep\n          svscan-1678  [000] .... 6412453.704049: do_nanosleep <-hrtimer_nanosleep\n          vmstat-28371 [000] .... 6412454.216524: do_nanosleep <-hrtimer_nanosleep\n          vmstat-28371 [000] .... 6412455.216816: do_nanosleep <-hrtimer_nanosleep\n          vmstat-28371 [000] .... 6412456.217093: do_nanosleep <-hrtimer_nanosleep\n          vmstat-28371 [000] .... 6412457.217378: do_nanosleep <-hrtimer_nanosleep\n          vmstat-28371 [000] .... 6412458.217660: do_nanosleep <-hrtimer_nanosleep\n^C\nEnding tracing...\n\nWhile tracing, I ran a \"vmstat 1\" in another window. vmstat and its process ID\ncan be seen as the 1st column, and the timestamp and one second intervals can\nbe seen as the 4th column.\n\nThis is basic details: who was on-CPU (process name and PID), flags, timestamp,\nand calling function. Treat this as the next step, after funccount, for getting\na little more information on kernel function execution, before using more\ncapabilities to dig further.\n\nThis is Linux 3.16, and the output is the ftrace text buffer format, which has\nchanged slightly between kernel versions.\n\n\nTo see the column headers, use -H. This is Linux 3.16:\n\n# ./functrace -H do_nanosleep\nTracing \"do_nanosleep\"... Ctrl-C to end.\n# tracer: function\n#\n# entries-in-buffer/entries-written: 0/0   #P:2\n#\n#                              _-----=> irqs-off\n#                             / _----=> need-resched\n#                            | / _---=> hardirq/softirq\n#                            || / _--=> preempt-depth\n#                            ||| /     delay\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#              | |       |   ||||       |         |\n          svscan-1678  [001] .... 6413283.729520: do_nanosleep <-hrtimer_nanosleep\n          svscan-1678  [001] .... 6413288.729679: do_nanosleep <-hrtimer_nanosleep\n\nFor comparison, here's Linux 3.2:\n\n# ./functrace -H do_nanosleep\nTracing \"do_nanosleep\"... Ctrl-C to end.\n# tracer: function\n#\n#           TASK-PID    CPU#    TIMESTAMP  FUNCTION\n#              | |       |          |         |\n          vmstat-11789 [000] 1763207.021204: do_nanosleep <-hrtimer_nanosleep\n          vmstat-11789 [000] 1763208.022970: do_nanosleep <-hrtimer_nanosleep\n          vmstat-11789 [000] 1763209.023267: do_nanosleep <-hrtimer_nanosleep\n\nFor documentation on the exact format, see the Linux kernel source under\nDocumentation/trace/ftrace.txt.\n\n\nThis error:\n\n# ./functrace 'ext4_z*'\nTracing \"ext4_z*\"... Ctrl-C to end.\n./functrace: line 136: echo: write error: Invalid argument\nERROR: enabling \"ext4_z*\". Exiting.\n\nIs because there were no functions beginning with \"ext4_z\". You can check\navailable functions in the /sys/kernel/debug/tracing/available_filter_functions\nfile.\n\n\nYou might want to use funccount to check the frequency of events before using\nfunctrace. For example, counting ext3 events on a system:\n\n# ./funccount -d 10 'ext3*'\nTracing \"ext3*\" for 10 seconds...\n\nFUNC                              COUNT\next3_journal_dirty_data               1\next3_ordered_write_end                1\next3_write_begin                      1\next3_writepage_trans_blocks           1\next3_dirty_inode                      2\next3_do_update_inode                  2\next3_get_group_desc                   2\next3_get_inode_block.isra.20          2\next3_get_inode_flags                  2\next3_get_inode_loc                    2\next3_mark_iloc_dirty                  2\next3_mark_inode_dirty                 2\next3_reserve_inode_write              2\next3_journal_start_sb                 3\next3_block_to_path.isra.22            6\next3_bmap                             6\next3_get_block                        6\next3_get_blocks_handle                6\next3_get_branch                       6\next3_discard_reservation             11\next3_ioctl                           11\next3_release_file                    11\n\nEnding tracing...\n\nDuring 10 seconds, there weren't many ext3 calls. I might consider tracing\nthem all (warnings about dynamic tracing many kernel functions apply: test\nbefore use, as in the past there have been bugs causing panics).\n\n# ./functrace 'ext3_*'\nTracing \"ext3_*\"... Ctrl-C to end.\n register_start.-17008 [000] 1763557.577985: ext3_release_file <-__fput\n register_start.-17008 [000] 1763557.577987: ext3_discard_reservation <-ext3_release_file\n register_start.-17026 [000] 1763558.163620: ext3_ioctl <-file_ioctl\n register_start.-17026 [000] 1763558.481081: ext3_release_file <-__fput\n register_start.-17026 [000] 1763558.481083: ext3_discard_reservation <-ext3_release_file\n register_start.-17041 [000] 1763559.186984: ext3_ioctl <-file_ioctl\n register_start.-17041 [000] 1763559.511267: ext3_release_file <-__fput\n[...]\n\nFor comparison, here's a different system and ext4:\n\n# ./funccount -d 10 'ext4*'\nTracing \"ext4*\" for 10 seconds...\n\nFUNC                              COUNT\next4_journal_commit_callback          2\next4_htree_fill_tree                  6\next4_htree_free_dir_info              6\next4_release_dir                      6\next4_readdir                         12\next4fs_dirhash                       29\next4_htree_store_dirent              29\next4_follow_link                     36\next4_file_mmap                       42\next4_free_data_callback              44\next4_getattr                         45\next4_bmap                            62\next4_get_block                       62\next4_add_entry                      280\next4_add_nondir                     280\next4_alloc_da_blocks                280\next4_alloc_inode                    280\next4_bio_write_page                 280\next4_can_truncate                   280\next4_claim_free_clusters            280\next4_clear_inode                    280\next4_create                         280\next4_da_get_block_prep              280\next4_da_invalidatepage              280\next4_da_update_reserve_space        280\next4_da_write_begin                 280\next4_da_write_end                   280\next4_dec_count.isra.22              280\next4_delete_entry                   280\next4_destroy_inode                  280\next4_drop_inode                     280\next4_end_bio                        280\next4_es_init_tree                   280\next4_es_lru_del                     280\next4_evict_inode                    280\next4_ext_calc_metadata_amount       280\next4_ext_correct_indexes            280\next4_ext_find_goal                  280\next4_ext_insert_extent              280\next4_ext_remove_space               280\next4_ext_tree_init                  280\next4_ext_truncate                   280\next4_ext_truncate_extend_resta      280\next4_ext_try_to_merge               280\next4_ext_try_to_merge_right         280\next4_file_write_iter                280\next4_find_dest_de                   280\next4_finish_bio                     280\next4_free_blocks                    280\next4_free_inode                     280\next4_generic_delete_entry           280\next4_has_free_clusters              280\next4_i_callback                     280\next4_init_acl                       280\next4_init_security                  280\next4_inode_attach_jinode            280\next4_inode_to_goal_block            280\next4_insert_dentry                  280\next4_invalidatepage                 280\next4_io_submit_init                 280\next4_itable_unused_count            280\next4_lookup                         280\next4_mb_complex_scan_group          280\next4_mb_find_by_goal                280\next4_mb_free_metadata               280\next4_mb_initialize_context          280\next4_mb_mark_diskspace_used         280\next4_mb_new_blocks                  280\next4_mb_normalize_request           280\next4_mb_regular_allocator           280\next4_mb_release_context             280\next4_mb_use_best_found              280\next4_mb_use_preallocated            280\next4_nonda_switch                   280\next4_orphan_del                     280\next4_put_io_end_defer               280\next4_releasepage                    280\next4_rename                         280\next4_set_aops                       280\next4_setent                         280\next4_set_inode_flags                280\next4_truncate                       280\next4_writepages                     280\next4_writepage_trans_blocks         280\next4_xattr_delete_inode             280\next4_xattr_get                      285\next4_xattr_ibody_get                285\next4_xattr_security_get             285\next4_bread                          286\next4_release_file                   288\next4_file_open                      305\next4_superblock_csum_set            494\next4_block_bitmap_csum_set          560\next4_es_free_extent                 560\next4_es_insert_extent               560\next4_es_remove_extent               560\next4_ext_find_extent                560\next4_ext_map_blocks                 560\next4_free_group_clusters_set        560\next4_free_inodes_set                560\next4_get_group_no_and_offset        560\next4_get_reserved_space             560\next4_init_io_end                    560\next4_inode_bitmap_csum_set          560\next4_io_submit                      560\next4_mb_good_group                  560\next4_orphan_add                     560\next4_put_io_end                     560\next4_read_block_bitmap              560\next4_read_block_bitmap_nowait       560\next4_read_inode_bitmap              560\next4_release_io_end                 560\next4_set_bits                       560\next4_validate_block_bitmap          560\next4_wait_block_bitmap              560\next4_mb_load_buddy                  604\next4_mb_unload_buddy.isra.24        604\next4_block_bitmap                   840\next4_discard_preallocations         840\next4_ext_drop_refs                  840\next4_ext_get_access.isra.30         840\next4_ext_index_trans_blocks         840\next4_find_entry                     840\next4_free_group_clusters            840\next4_handle_dirty_dirent_node       840\next4_inode_bitmap                   840\next4_meta_trans_blocks              840\next4_dirty_inode                    845\next4_free_inodes_count             1120\next4_group_desc_csum               1120\next4_group_desc_csum_set           1120\next4_getblk                        1126\next4_map_blocks                    1468\next4_es_lookup_extent              1748\next4_mb_check_limits               1875\next4_es_lru_add                    2028\next4_data_block_valid              2308\next4_journal_check_start           3085\next4_mark_inode_dirty              5325\next4_get_inode_flags               5951\next4_get_inode_loc                 5951\next4_mark_iloc_dirty               5951\next4_reserve_inode_write           5951\next4_inode_table                   7071\next4_get_group_desc                8471\next4_has_inline_data               9486\n\nEnding tracing...\n\nThere are many functions called frequently. Tracing them all may cost\nsignificant performance overhead. I may read through this list and look for\nthe most interesting functions to trace, reducing overheads by only selecting\na few.\n\nFor example, ext4_create() looks interesting:\n\n# ./functrace ext4_create\nTracing \"ext4_create\"... Ctrl-C to end.\n       supervise-1681  [000] .... 6414396.700163: ext4_create <-vfs_create\n       supervise-1684  [001] .... 6414396.700287: ext4_create <-vfs_create\n       supervise-1681  [000] .... 6414396.700598: ext4_create <-vfs_create\n       supervise-1684  [001] .... 6414396.700636: ext4_create <-vfs_create\n       supervise-1687  [001] .... 6414396.701577: ext4_create <-vfs_create\n       supervise-1688  [000] .... 6414396.702590: ext4_create <-vfs_create\n       supervise-1693  [001] .... 6414396.702829: ext4_create <-vfs_create\n       supervise-1693  [001] .... 6414396.703592: ext4_create <-vfs_create\n       supervise-1688  [000] .... 6414396.703598: ext4_create <-vfs_create\n       supervise-1687  [001] .... 6414396.703988: ext4_create <-vfs_create\n       supervise-1685  [001] .... 6414396.704126: ext4_create <-vfs_create\n       supervise-1685  [001] .... 6414396.704458: ext4_create <-vfs_create\n       supervise-1682  [001] .... 6414396.704577: ext4_create <-vfs_create\n       supervise-1683  [000] .... 6414396.704984: ext4_create <-vfs_create\n       supervise-1682  [001] .... 6414396.704985: ext4_create <-vfs_create\n[...]\n\nNow I know that different PIDs of the supervise program are calling ext4_create,\nof around the same time, and from vfs_create().\n\n\nThe duration mode uses buffering, instead of printing events as they occur.\nThis greatly reduces overheads. For example:\n\n# ./functrace -d 10 ext4_create > out.ext4_create\n# wc out.ext4_create\n  283  1687 21059 out.ext4_create\n\nNote that the buffer has a limited size. Check the timestamps to see if the\nrange does not match your duration, as one clue that the buffer was exhausted\nand events were missed.\n\n\nUse -h to print the USAGE message:\n\n# ./functrace -h\nUSAGE: functrace [-hH] [-p PID] [-L TID] [-d secs] funcstring\n                 -d seconds      # trace duration, and use buffers\n                 -h              # this usage message\n                 -H              # include column headers\n                 -p PID          # trace when this pid is on-CPU\n                 -L TID          # trace when this thread is on-CPU\n  eg,\n       functrace do_nanosleep    # trace the do_nanosleep() function\n       functrace '*sleep'        # trace functions ending in \"sleep\"\n       functrace -p 198 'vfs*'   # trace \"vfs*\" funcs for PID 198\n       functrace 'tcp*' > out    # trace all \"tcp*\" funcs to out file\n       functrace -d 1 'tcp*' > out  # trace 1 sec, then write out file\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/iolatency_example.txt",
    "content": "Demonstrations of iolatency, the Linux ftrace version.\n\n\nHere's a busy system doing over 4k disk IOPS:\n\n# ./iolatency\nTracing block I/O. Output every 1 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 4381     |######################################|\n       1 -> 2       : 9        |#                                     |\n       2 -> 4       : 5        |#                                     |\n       4 -> 8       : 0        |                                      |\n       8 -> 16      : 1        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 4053     |######################################|\n       1 -> 2       : 18       |#                                     |\n       2 -> 4       : 9        |#                                     |\n       4 -> 8       : 2        |#                                     |\n       8 -> 16      : 1        |#                                     |\n      16 -> 32      : 1        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 4658     |######################################|\n       1 -> 2       : 9        |#                                     |\n       2 -> 4       : 2        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 4298     |######################################|\n       1 -> 2       : 17       |#                                     |\n       2 -> 4       : 10       |#                                     |\n       4 -> 8       : 1        |#                                     |\n       8 -> 16      : 1        |#                                     |\n^C\nEnding tracing...\n\nDisk I/O latency is usually between 0 and 1 milliseconds, as this system uses\nSSDs. There are occasional outliers, up to the 16->32 ms range.\n\nIdentifying outliers like these is difficult from iostat(1) alone, which at\nthe same time reported:\n\n# iostat 1\n[...]\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\n           0.53    0.00    1.05   46.84    0.53   51.05\n\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\nxvdap1            0.00     0.00    0.00   28.00     0.00   112.00     8.00     0.02    0.71    0.00    0.71   0.29   0.80\nxvdb              0.00     0.00 2134.00    0.00 18768.00     0.00    17.59     0.51    0.24    0.24    0.00   0.23  50.00\nxvdc              0.00     0.00 2088.00    0.00 18504.00     0.00    17.72     0.47    0.22    0.22    0.00   0.22  46.40\nmd0               0.00     0.00 4222.00    0.00 37256.00     0.00    17.65     0.00    0.00    0.00    0.00   0.00   0.00\n\nI/O latency (\"await\") averages 0.24 and 0.22 ms for our busy disks, but this\noutput doesn't show that occasionally is much higher.\n\nTo get more information on these I/O, try the iosnoop(8) tool.\n\n\nThe -Q option includes the block I/O queued time, by tracing based on\nblock_rq_insert instead of block_rq_issue:\n\n# ./iolatency -Q\nTracing block I/O. Output every 1 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1913     |######################################|\n       1 -> 2       : 438      |#########                             |\n       2 -> 4       : 100      |##                                    |\n       4 -> 8       : 145      |###                                   |\n       8 -> 16      : 43       |#                                     |\n      16 -> 32      : 43       |#                                     |\n      32 -> 64      : 1        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 2360     |######################################|\n       1 -> 2       : 132      |###                                   |\n       2 -> 4       : 72       |##                                    |\n       4 -> 8       : 14       |#                                     |\n       8 -> 16      : 1        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 2138     |######################################|\n       1 -> 2       : 496      |#########                             |\n       2 -> 4       : 81       |##                                    |\n       4 -> 8       : 40       |#                                     |\n       8 -> 16      : 1        |#                                     |\n      16 -> 32      : 2        |#                                     |\n^C\nEnding tracing...\n\nI use this along with the default mode to identify problems of load (queueing)\nvs problems of the device, which is shown by default.\n\n\nHere's a more interesting system. This is doing a mixed read/write workload,\nand has a pretty awful latency distribution:\n\n# ./iolatency 5 3\nTracing block I/O. Output every 5 seconds.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 2809     |######################################|\n       1 -> 2       : 32       |#                                     |\n       2 -> 4       : 14       |#                                     |\n       4 -> 8       : 6        |#                                     |\n       8 -> 16      : 7        |#                                     |\n      16 -> 32      : 14       |#                                     |\n      32 -> 64      : 39       |#                                     |\n      64 -> 128     : 1556     |######################                |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 3027     |######################################|\n       1 -> 2       : 19       |#                                     |\n       2 -> 4       : 6        |#                                     |\n       4 -> 8       : 5        |#                                     |\n       8 -> 16      : 3        |#                                     |\n      16 -> 32      : 7        |#                                     |\n      32 -> 64      : 14       |#                                     |\n      64 -> 128     : 540      |#######                               |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 2939     |######################################|\n       1 -> 2       : 25       |#                                     |\n       2 -> 4       : 15       |#                                     |\n       4 -> 8       : 2        |#                                     |\n       8 -> 16      : 3        |#                                     |\n      16 -> 32      : 7        |#                                     |\n      32 -> 64      : 17       |#                                     |\n      64 -> 128     : 936      |#############                         |\n\nEnding tracing...\n\nIt's multi-modal, with most I/O taking 0 to 1 milliseconds, then many between\n64 and 128 milliseconds. This is how it looks in iostat:\n\n# iostat -x 1\n\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\n           0.52    0.00   12.37   32.99    0.00   54.12\n\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\nxvdap1            0.00    12.00    0.00  156.00     0.00 19968.00   256.00    52.17  184.38    0.00  184.38   2.33  36.40\nxvdb              0.00     0.00  298.00    0.00  2732.00     0.00    18.34     0.04    0.12    0.12    0.00   0.11   3.20\nxvdc              0.00     0.00  297.00    0.00  2712.00     0.00    18.26     0.08    0.27    0.27    0.00   0.24   7.20\nmd0               0.00     0.00  595.00    0.00  5444.00     0.00    18.30     0.00    0.00    0.00    0.00   0.00   0.00\n\nFortunately, it turns out that the high latency is to xvdap1, which is for files\nfrom a low priority application (processing and writing log files). A high\npriority application is reading from the other disks, xvdb and xvdc.\n\nExamining xvdap1 only:\n\n# ./iolatency -d 202,1 5\nTracing block I/O. Output every 5 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 38       |##                                    |\n       1 -> 2       : 18       |#                                     |\n       2 -> 4       : 0        |                                      |\n       4 -> 8       : 0        |                                      |\n       8 -> 16      : 5        |#                                     |\n      16 -> 32      : 11       |#                                     |\n      32 -> 64      : 26       |##                                    |\n      64 -> 128     : 894      |######################################|\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 75       |###                                   |\n       1 -> 2       : 11       |#                                     |\n       2 -> 4       : 0        |                                      |\n       4 -> 8       : 4        |#                                     |\n       8 -> 16      : 4        |#                                     |\n      16 -> 32      : 7        |#                                     |\n      32 -> 64      : 13       |#                                     |\n      64 -> 128     : 1141     |######################################|\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 61       |########                              |\n       1 -> 2       : 21       |###                                   |\n       2 -> 4       : 5        |#                                     |\n       4 -> 8       : 1        |#                                     |\n       8 -> 16      : 5        |#                                     |\n      16 -> 32      : 7        |#                                     |\n      32 -> 64      : 19       |###                                   |\n      64 -> 128     : 324      |######################################|\n     128 -> 256     : 7        |#                                     |\n     256 -> 512     : 26       |####                                  |\n^C\nEnding tracing...\n\nAnd now xvdb:\n\n# ./iolatency -d 202,16 5\nTracing block I/O. Output every 5 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1427     |######################################|\n       1 -> 2       : 5        |#                                     |\n       2 -> 4       : 3        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1409     |######################################|\n       1 -> 2       : 6        |#                                     |\n       2 -> 4       : 1        |#                                     |\n       4 -> 8       : 1        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1478     |######################################|\n       1 -> 2       : 6        |#                                     |\n       2 -> 4       : 5        |#                                     |\n       4 -> 8       : 0        |                                      |\n       8 -> 16      : 2        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1437     |######################################|\n       1 -> 2       : 5        |#                                     |\n       2 -> 4       : 7        |#                                     |\n       4 -> 8       : 0        |                                      |\n       8 -> 16      : 1        |#                                     |\n[...]\n\nWhile that's much better, it is reaching the 8 - 16 millisecond range,\nand these are SSDs with a light workload (~1500 IOPS).\n\nI already know from iosnoop(8) analysis the reason for these high latency\noutliers: they are queued behind writes. However, these writes are to a\ndifferent disk -- somewhere in this virtualized guest (Xen) there may be a\nshared I/O queue.\n\nOne way to explore this is to reduce the queue length for the low priority disk,\nso that it is less likely to pollute any shared queue. (There are other ways to\ninvestigate and fix this too.) Here I reduce the disk queue length from its\ndefault of 128 to 4:\n\n# echo 4 > /sys/block/xvda1/queue/nr_requests\n\nThe overall distribution looks much better:\n\n# ./iolatency 5\nTracing block I/O. Output every 5 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 3005     |######################################|\n       1 -> 2       : 19       |#                                     |\n       2 -> 4       : 9        |#                                     |\n       4 -> 8       : 45       |#                                     |\n       8 -> 16      : 859      |###########                           |\n      16 -> 32      : 16       |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 2959     |######################################|\n       1 -> 2       : 43       |#                                     |\n       2 -> 4       : 16       |#                                     |\n       4 -> 8       : 39       |#                                     |\n       8 -> 16      : 1009     |#############                         |\n      16 -> 32      : 76       |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 3031     |######################################|\n       1 -> 2       : 27       |#                                     |\n       2 -> 4       : 9        |#                                     |\n       4 -> 8       : 24       |#                                     |\n       8 -> 16      : 422      |######                                |\n      16 -> 32      : 5        |#                                     |\n^C\nEnding tracing...\n\nLatency only reaching 32 ms.\n\nOur important disk didn't appear to change much -- maybe a slight improvement\nto the outliers:\n\n# ./iolatency -d 202,16 5\nTracing block I/O. Output every 5 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1449     |######################################|\n       1 -> 2       : 6        |#                                     |\n       2 -> 4       : 5        |#                                     |\n       4 -> 8       : 1        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1519     |######################################|\n       1 -> 2       : 12       |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1466     |######################################|\n       1 -> 2       : 2        |#                                     |\n       2 -> 4       : 3        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 1460     |######################################|\n       1 -> 2       : 4        |#                                     |\n       2 -> 4       : 7        |#                                     |\n[...]\n\nAnd here's the other disk after the queue length change:\n\n# ./iolatency -d 202,1 5\nTracing block I/O. Output every 5 seconds. Ctrl-C to end.\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 85       |###                                   |\n       1 -> 2       : 12       |#                                     |\n       2 -> 4       : 21       |#                                     |\n       4 -> 8       : 76       |##                                    |\n       8 -> 16      : 1539     |######################################|\n      16 -> 32      : 10       |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 123      |##################                    |\n       1 -> 2       : 8        |##                                    |\n       2 -> 4       : 6        |#                                     |\n       4 -> 8       : 17       |###                                   |\n       8 -> 16      : 270      |######################################|\n      16 -> 32      : 2        |#                                     |\n\n  >=(ms) .. <(ms)   : I/O      |Distribution                          |\n       0 -> 1       : 91       |###                                   |\n       1 -> 2       : 23       |#                                     |\n       2 -> 4       : 8        |#                                     |\n       4 -> 8       : 71       |###                                   |\n       8 -> 16      : 1223     |######################################|\n      16 -> 32      : 12       |#                                     |\n^C\nEnding tracing...\n\nMuch better looking distribution.\n\n\nUse -h to print the USAGE message:\n\n# ./iolatency -h\nUSAGE: iolatency [-hQT] [-d device] [-i iotype] [interval [count]]\n                 -d device       # device string (eg, \"202,1)\n                 -i iotype       # match type (eg, '*R*' for all reads)\n                 -Q              # use queue insert as start time\n                 -T              # timestamp on output\n                 -h              # this usage message\n                 interval        # summary interval, seconds (default 1)\n                 count           # number of summaries\n  eg,\n       iolatency                 # summarize latency every second\n       iolatency -Q              # include block I/O queue time\n       iolatency 5 2             # 2 x 5 second summaries\n       iolatency -i '*R*'        # trace reads\n       iolatency -d 202,1        # trace device 202,1 only\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/iosnoop_example.txt",
    "content": "Demonstrations of iosnoop, the Linux ftrace version.\n\n\nHere's Linux 3.16, tracing tar archiving a filesystem:\n\n# ./iosnoop \nTracing block I/O... Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\nsupervise        1809   W    202,1    17039968     4096       1.32\nsupervise        1809   W    202,1    17039976     4096       1.30\ntar              14794  RM   202,1    8457608      4096       7.53\ntar              14794  RM   202,1    8470336      4096      14.90\ntar              14794  RM   202,1    8470368      4096       0.27\ntar              14794  RM   202,1    8470784      4096       7.74\ntar              14794  RM   202,1    8470360      4096       0.25\ntar              14794  RM   202,1    8469968      4096       0.24\ntar              14794  RM   202,1    8470240      4096       0.24\ntar              14794  RM   202,1    8470392      4096       0.23\ntar              14794  RM   202,1    8470544      4096       5.96\ntar              14794  RM   202,1    8470552      4096       0.27\ntar              14794  RM   202,1    8470384      4096       0.24\n[...]\n\nThe \"tar\" I/O looks like it is slightly random (based on BLOCK) and 4 Kbytes\nin size (BYTES). One returned in 14.9 milliseconds, but the rest were fast,\nso fast (0.24 ms) some may be returning from some level of cache (disk or\ncontroller).\n\nThe \"RM\" TYPE means Read of Metadata. The start of the trace shows a\ncouple of Writes by supervise PID 1809.\n\n\nHere's a deliberate random I/O workload:\n\n# ./iosnoop \nTracing block I/O. Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\nrandread         9182   R    202,32   30835224     8192       0.18\nrandread         9182   R    202,32   21466088     8192       0.15\nrandread         9182   R    202,32   13529496     8192       0.16\nrandread         9182   R    202,16   21250648     8192       0.18\nrandread         9182   R    202,16   1536776      32768      0.30\nrandread         9182   R    202,32   17157560     24576      0.23\nrandread         9182   R    202,32   21313320     8192       0.16\nrandread         9182   R    202,32   862184       8192       0.18\nrandread         9182   R    202,16   25496872     8192       0.21\nrandread         9182   R    202,32   31471768     8192       0.18\nrandread         9182   R    202,16   27571336     8192       0.20\nrandread         9182   R    202,16   30783448     8192       0.16\nrandread         9182   R    202,16   21435224     8192       1.28\nrandread         9182   R    202,16   970616       8192       0.15\nrandread         9182   R    202,32   13855608     8192       0.16\nrandread         9182   R    202,32   17549960     8192       0.15\nrandread         9182   R    202,32   30938232     8192       0.14\n[...]\n\nNote the changing offsets. The resulting latencies are very good in this case,\nbecause the storage devices are flash memory-based solid state disks (SSDs).\nFor rotational disks, I'd expect these latencies to be roughly 10 ms.\n\n\nHere's an idle Linux 3.2 system:\n\n# ./iosnoop \nTracing block I/O. Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\nsupervise        3055   W    202,1    12852496     4096       0.64\nsupervise        3055   W    202,1    12852504     4096       1.32\nsupervise        3055   W    202,1    12852800     4096       0.55\nsupervise        3055   W    202,1    12852808     4096       0.52\njbd2/xvda1-212   212    WS   202,1    1066720      45056     41.52\njbd2/xvda1-212   212    WS   202,1    1066808      12288     41.52\njbd2/xvda1-212   212    WS   202,1    1066832      4096      32.37\nsupervise        3055   W    202,1    12852800     4096      14.28\nsupervise        3055   W    202,1    12855920     4096      14.07\nsupervise        3055   W    202,1    12855960     4096       0.67\nsupervise        3055   W    202,1    12858208     4096       1.00\nflush:1-409      409    W    202,1    12939640     12288     18.00\n[...]\n\nThis shows supervise doing various writes from PID 3055. The highest latency\nwas from jbd2/xvda1-212, the journaling block device driver, doing\nsynchronous writes (TYPE = WS).\n\n\nOptions can be added to show the start time (-s) and end time (-t):\n\n# ./iosnoop -ts\nTracing block I/O. Ctrl-C to end.\nSTARTs         ENDs           COMM             PID    TYPE DEV      BLOCK        BYTES     LATms\n5982800.302061 5982800.302679 supervise        1809   W    202,1    17039600     4096       0.62\n5982800.302423 5982800.302842 supervise        1809   W    202,1    17039608     4096       0.42\n5982800.304962 5982800.305446 supervise        1801   W    202,1    17039616     4096       0.48\n5982800.305250 5982800.305676 supervise        1801   W    202,1    17039624     4096       0.43\n5982800.308849 5982800.309452 supervise        1810   W    202,1    12862464     4096       0.60\n5982800.308856 5982800.309470 supervise        1806   W    202,1    17039632     4096       0.61\n5982800.309206 5982800.309740 supervise        1806   W    202,1    17039640     4096       0.53\n5982800.309211 5982800.309805 supervise        1810   W    202,1    12862472     4096       0.59\n5982800.309332 5982800.309953 supervise        1812   W    202,1    17039648     4096       0.62\n5982800.309676 5982800.310283 supervise        1812   W    202,1    17039656     4096       0.61\n[...]\n\nThis is useful when gathering I/O event data for post-processing.\n\n\nNow for matching on a single PID:\n\n# ./iosnoop -p 1805\nTracing block I/O issued by PID 1805. Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\nsupervise        1805   W    202,1    17039648     4096       0.68\nsupervise        1805   W    202,1    17039672     4096       0.60\nsupervise        1805   W    202,1    17040040     4096       0.62\nsupervise        1805   W    202,1    17040056     4096       0.47\nsupervise        1805   W    202,1    17040624     4096       0.49\nsupervise        1805   W    202,1    17040632     4096       0.44\n^C\nEnding tracing...\n\nThis option works by using an in-kernel filter for that PID on I/O issue. There\nis also a \"-n\" option to match on process names, however, that currently does so\nin user space, so is less efficient.\n\nI would say that this will generally identify the origin process, but there will\nbe an error margin. Depending on the file system, block I/O queueing, and I/O\nsubsystem, this could miss events that aren't issued in this PID context but are\nrelated to this PID (eg, triggering a read readahead on the completion of\nprevious I/O. Again, whether this happens is up to the file system and storage\nsubsystem). You can try the -Q option for more reliable process identification.\n\n\nThe -Q option begins tracing on block I/O queue insert, instead of issue.\nHere's before and after, while dd(1) writes a large file:\n\n# ./iosnoop\nTracing block I/O. Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\ndd               26983  WS   202,16   4064416      45056     16.70\ndd               26983  WS   202,16   4064504      45056     16.72\ndd               26983  WS   202,16   4064592      45056     16.74\ndd               26983  WS   202,16   4064680      45056     16.75\ncat              27031  WS   202,16   4064768      45056     16.56\ncat              27031  WS   202,16   4064856      45056     16.46\ncat              27031  WS   202,16   4064944      45056     16.40\ngawk             27030  WS   202,16   4065032      45056      0.88\ngawk             27030  WS   202,16   4065120      45056      1.01\ngawk             27030  WS   202,16   4065208      45056     16.15\ngawk             27030  WS   202,16   4065296      45056     16.16\ngawk             27030  WS   202,16   4065384      45056     16.16\n[...]\n\nThe output here shows the block I/O time from issue to completion (LATms),\nwhich is largely representative of the device.\n\nThe process names and PIDs identify dd, cat, and gawk. By default iosnoop shows\nwho is on-CPU at time of block I/O issue, but these may not be the processes\nthat originated the I/O. In this case (having debugged it), the reason is that\nprocesses such as cat and gawk are making hypervisor calls (this is a Xen\nguest instance), eg, for memory operations, and during hypervisor processing a\nqueue of pending work is checked and dispatched. So cat and gawk were on-CPU\nwhen the block device I/O was issued, but they didn't originate it.\n\nNow the -Q option is used:\n\n# ./iosnoop -Q\nTracing block I/O. Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\nkjournald        1217   WS   202,16   6132200      45056    141.12\nkjournald        1217   WS   202,16   6132288      45056    141.10\nkjournald        1217   WS   202,16   6132376      45056    141.10\nkjournald        1217   WS   202,16   6132464      45056    141.11\nkjournald        1217   WS   202,16   6132552      40960    141.11\ndd               27718  WS   202,16   6132624      4096       0.18\nflush:16-1279    1279   W    202,16   6132632      20480      0.52\nflush:16-1279    1279   W    202,16   5940856      4096       0.50\nflush:16-1279    1279   W    202,16   5949056      4096       0.52\nflush:16-1279    1279   W    202,16   5957256      4096       0.54\nflush:16-1279    1279   W    202,16   5965456      4096       0.56\nflush:16-1279    1279   W    202,16   5973656      4096       0.58\nflush:16-1279    1279   W    202,16   5981856      4096       0.60\nflush:16-1279    1279   W    202,16   5990056      4096       0.63\n[...]\n\nThis uses the block_rq_insert tracepoint as the starting point of I/O, instead\nof block_rq_issue. This makes the following differences to columns and options:\n\n- COMM: more likely to show the originating process.\n- PID: more likely to show the originating process.\n- LATms: shows the I/O time, including time spent on the block I/O queue.\n- STARTs (not shown above): shows the time of queue insert, not I/O issue.\n- -p PID: more likely to match the originating process.\n- -n name: more likely to match the originating process.\n\nThe reason that this ftrace-based iosnoop does not just instrument both insert\nand issue tracepoints is one of overhead. Even with buffering, iosnoop can\nhave difficulty under high load.\n\n\nIf I want to capture events for post-processing, I use the duration mode, which\nnot only lets me set the duration, but also uses buffering, which reduces the\noverheads of tracing.\n\nCapturing 5 seconds, with both start timestamps (-s) and end timestamps (-t):\n\n# time ./iosnoop -ts 5 > out\n\nreal\t0m5.566s\nuser\t0m0.336s\nsys\t0m0.140s\n# wc out\n  27010  243072 2619744 out\n\nThis server is doing over 5,000 disk IOPS. Even with buffering, this did\nconsume a measurable amount of CPU to capture: 0.48 seconds of CPU time in\ntotal. Note that the run took 5.57 seconds: this is 5 seconds for the capture,\nfollowed by the CPU time for iosnoop to fetch and process the buffer.\n\nNow tracing for 30 seconds:\n\n# time ./iosnoop -ts 30 > out\n\nreal\t0m31.207s\nuser\t0m0.884s\nsys\t0m0.472s\n# wc out\n  64259  578313 6232898 out\n\nSince it's the same server and workload, this should have over 150k events,\nbut only has 64k. The tracing buffer has overflowed, and events have been\ndropped. If I really must capture this many events, I can either increase\nthe trace buffer size (it's the bufsize_kb setting in the script), or, use\na different tracer (perf_evets, SystemTap, ktap, etc.) If the IOPS rate is low\n(eg, less than 5k), then unbuffered (no duration), despite the higher overheads,\nmay be sufficient, and will keep capturing events until Ctrl-C.\n\n\nHere's an example of digging into the sequence of I/O to explain an outlier.\nMy randread program on an SSD server (which is an AWS EC2 instance) usually\nexperiences about 0.15 ms I/O latency, but there are some outliers as high as\n20 milliseconds. Here's an excerpt:\n\n# ./iosnoop -ts > out\n# more out\nTracing block I/O. Ctrl-C to end.\nSTARTs         ENDs           COMM             PID    TYPE DEV      BLOCK        BYTES     LATms\n6037559.121523 6037559.121685 randread         22341  R    202,32   29295416     8192       0.16\n6037559.121719 6037559.121874 randread         22341  R    202,16   27515304     8192       0.16\n[...]\n6037595.999508 6037596.000051 supervise        1692   W    202,1    12862968     4096       0.54\n6037595.999513 6037596.000144 supervise        1687   W    202,1    17040160     4096       0.63\n6037595.999634 6037596.000309 supervise        1693   W    202,1    17040168     4096       0.68\n6037595.999937 6037596.000440 supervise        1693   W    202,1    17040176     4096       0.50\n6037596.000579 6037596.001192 supervise        1689   W    202,1    17040184     4096       0.61\n6037596.000826 6037596.001360 supervise        1689   W    202,1    17040192     4096       0.53\n6037595.998302 6037596.018133 randread         22341  R    202,32   954168       8192      20.03\n6037595.998303 6037596.018150 randread         22341  R    202,32   954200       8192      20.05\n6037596.018182 6037596.018347 randread         22341  R    202,32   18836600     8192       0.16\n[...]\n\nIt's important to sort on the I/O completion time (ENDs). In this case it's\nalready in the correct order.\n\nSo my 20 ms reads happened after a large group of supervise writes were\ncompleted (I truncated dozens of supervise write lines to keep this example\nshort). Other latency outliers in this output file showed the same sequence:\nslow reads after a batch of writes.\n\nNote the I/O request timestamp (STARTs), which shows that these 20 ms reads were\nissued before the supervise writes – so they had been sitting on a queue. I've\ndebugged this type of issue many times before, but this one is different: those\nwrites were to a different device (202,1), so I would have assumed they would be\non different queues, and wouldn't interfere with each other. Somewhere in this\nsystem (Xen guest) it looks like there is a shared queue. (Having just\ndiscovered this using iosnoop, I can't yet tell you which queue, but I'd hope\nthat after identifying it there would be a way to tune its queueing behavior,\nso that we can eliminate or reduce the severity of these outliers.)\n\n\nUse -h to print the USAGE message:\n\n# ./iosnoop -h\nUSAGE: iosnoop [-hQst] [-d device] [-i iotype] [-p PID] [-n name]\n               [duration]\n                 -d device       # device string (eg, \"202,1)\n                 -i iotype       # match type (eg, '*R*' for all reads)\n                 -n name         # process name to match on I/O issue\n                 -p PID          # PID to match on I/O issue\n                 -Q              # use queue insert as start time\n                 -s              # include start time of I/O (s)\n                 -t              # include completion time of I/O (s)\n                 -h              # this usage message\n                 duration        # duration seconds, and use buffers\n  eg,\n       iosnoop                   # watch block I/O live (unbuffered)\n       iosnoop 1                 # trace 1 sec (buffered)\n       iosnoop -Q                # include queueing time in LATms\n       iosnoop -ts               # include start and end timestamps\n       iosnoop -i '*R*'          # trace reads\n       iosnoop -p 91             # show I/O issued when PID 91 is on-CPU\n       iosnoop -Qp 91            # show I/O queued by PID 91, queue time\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/killsnoop_example.txt",
    "content": "Demonstrations of killsnoop, the Linux ftrace version.\n\n\nWhat signals are happening on my system?\n\n# ./killsnoop \nTracing kill()s. Ctrl-C to end.\nCOMM             PID    TPID     SIGNAL     RETURN\npostgres         2209   2148     10         0   \npostgres         5416   2209     12         0   \npostgres         5416   2209     12         0   \nsupervise        2135   5465     15         0   \nsupervise        2135   5465     18         0   \n^C\nEnding tracing...\n\nThe first line of output shows that PID 2209, process name \"postgres\", has\nsent a signal 10 (SIGUSR1) to target PID 2148. This signal returned success (0).\n\nkilsnoop traces the kill() syscall, which is used to send signals to other\nprocesses. These signals can include SIGKILL and SIGTERM, both of which\nultimately kill the target process (in different fashions), but the signals\nmay also include other operations, including checking if a process still\nexists (signal 0). To read more about signals, see \"man -s7 signal\".\n\nkillsnoop can be useful to identify why some processes are abruptly and\nunexpectedly ending (also check for the OOM killer in dmesg).\n\n\nThe -s option can be used to print signal names instead of numbers:\n\n# ./killsnoop -s\nTracing kill()s. Ctrl-C to end.\nCOMM             PID    KILLED   SIGNAL     RETURN\npostgres         2209   2148     SIGUSR1    0   \npostgres         5665   2209     SIGUSR2    0   \npostgres         5665   2209     SIGUSR2    0   \nsupervise        2135   5711     SIGTERM    0   \nsupervise        2135   5711     SIGCONT    0   \nbash             27450  27450    0          0   \n[...]\n\nOn the last line: there wasn't a nice signal name for signal 0, so just numeric\n0 is printed. You'll see signal 0's used to check if processes still exist.\n\n\nUse -h to print the USAGE message:\n\n# ./opensnoop -h\nUSAGE: killsnoop [-ht] [-d secs] [-p PID] [-n name] [filename]\n                 -d seconds      # trace duration, and use buffers\n                 -n name         # process name to match \n                 -p PID          # PID to match on kill issue\n                 -t              # include time (seconds)\n                 -s              # human readable signal names\n                 -h              # this usage message\n  eg,\n       killsnoop                 # watch kill()s live (unbuffered)\n       killsnoop -d 1            # trace 1 sec (buffered)\n       killsnoop -p 181          # trace kill()s issued to PID 181 only\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/kprobe_example.txt",
    "content": "Demonstrations of kprobe, the Linux ftrace version.\n\n\nThis traces the kernel do_sys_open() function, when it is called:\n\n# ./kprobe p:do_sys_open\nTracing kprobe do_sys_open. Ctrl-C to end.\n          kprobe-26042 [001] d... 6910441.001452: do_sys_open: (do_sys_open+0x0/0x220)\n          kprobe-26042 [001] d... 6910441.001475: do_sys_open: (do_sys_open+0x0/0x220)\n          kprobe-26042 [001] d... 6910441.001866: do_sys_open: (do_sys_open+0x0/0x220)\n          kprobe-26042 [001] d... 6910441.001966: do_sys_open: (do_sys_open+0x0/0x220)\n       supervise-1689  [000] d... 6910441.083302: do_sys_open: (do_sys_open+0x0/0x220)\n       supervise-1693  [001] d... 6910441.083530: do_sys_open: (do_sys_open+0x0/0x220)\n       supervise-1689  [000] d... 6910441.083759: do_sys_open: (do_sys_open+0x0/0x220)\n       supervise-1693  [001] d... 6910441.083877: do_sys_open: (do_sys_open+0x0/0x220)\n[...]\n\nThe \"p:\" is for creating a probe. Use \"r:\" to probe the return of the function:\n\n# ./kprobe r:do_sys_open\nTracing kprobe do_sys_open. Ctrl-C to end.\n          kprobe-29475 [001] d... 6910688.229777: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n           <...>-29476 [001] d... 6910688.231101: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n           <...>-29476 [001] d... 6910688.231123: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n           <...>-29476 [001] d... 6910688.231530: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n           <...>-29476 [001] d... 6910688.231624: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n       supervise-1685  [001] d... 6910688.328776: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n       supervise-1689  [000] d... 6910688.328780: do_sys_open: (SyS_open+0x1e/0x20 <- do_sys_open)\n[...]\n\nThis output includes the function that the traced function is returning to.\n\n\nThe trace output can be a little different between kernel versions. Use -H to\nprint the header:\n\n# ./kprobe -H p:do_sys_open\nTracing kprobe do_sys_open. Ctrl-C to end.\n# tracer: nop\n#\n# entries-in-buffer/entries-written: 4/4   #P:2\n#\n#                              _-----=> irqs-off\n#                             / _----=> need-resched\n#                            | / _---=> hardirq/softirq\n#                            || / _--=> preempt-depth\n#                            ||| /     delay\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#              | |       |   ||||       |         |\n          kprobe-27952 [001] d... 6910580.008086: do_sys_open: (do_sys_open+0x0/0x220)\n          kprobe-27952 [001] d... 6910580.008109: do_sys_open: (do_sys_open+0x0/0x220)\n          kprobe-27952 [001] d... 6910580.008483: do_sys_open: (do_sys_open+0x0/0x220)\n[...]\n\nThese columns are explained in the kernel source under Documentation/trace/ftrace.txt.\n\n\nThis traces do_sys_open() returns, using a probe alias \"myopen\", and showing\nthe return value ($retval):\n\n# ./kprobe 'r:myopen do_sys_open $retval'\nTracing kprobe myopen. Ctrl-C to end.\n          kprobe-26386 [001] d... 6593278.858754: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x3\n           <...>-26387 [001] d... 6593278.860043: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x3\n           <...>-26387 [001] d... 6593278.860064: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x3\n           <...>-26387 [001] d... 6593278.860433: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x3\n           <...>-26387 [001] d... 6593278.860521: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x3\n       supervise-1685  [001] d... 6593279.178806: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1689  [001] d... 6593279.228756: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1689  [001] d... 6593279.229106: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1688  [000] d... 6593279.229501: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1695  [000] d... 6593279.229944: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1685  [001] d... 6593279.230104: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1687  [001] d... 6593279.230293: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1699  [000] d... 6593279.230381: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1692  [000] d... 6593279.230825: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1698  [000] d... 6593279.230915: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1698  [000] d... 6593279.231277: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n       supervise-1690  [000] d... 6593279.231703: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) arg1=0x9\n^C\nEnding tracing...\n\nThe string specified, 'r:myopen do_sys_open $retval', is a kprobe definition,\nand is the same as those documented in the Linux kernel source under\nDocumentation/trace/kprobetrace.txt, which can be written to the\n/sys/kernel/debug/tracing/kprobe_events file.\n\nApart from probe name aliases, you can also provide arbitrary names for\narguments. Eg, instead of the \"arg1\" default, calling it \"rval\":\n\n# ./kprobe 'r:myopen do_sys_open rval=$retval'\nTracing kprobe myopen. Ctrl-C to end.\n          kprobe-27454 [001] d... 6593356.250019: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x3\n           <...>-27455 [001] d... 6593356.251280: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x3\n           <...>-27455 [001] d... 6593356.251301: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x3\n           <...>-27455 [001] d... 6593356.251672: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x3\n           <...>-27455 [001] d... 6593356.251769: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x3\n       supervise-1689  [000] d... 6593356.859758: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x9\n       supervise-1689  [000] d... 6593356.860143: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x9\n       supervise-1696  [000] d... 6593356.862682: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x9\n       supervise-1685  [001] d... 6593356.862684: myopen: (SyS_open+0x1e/0x20 <- do_sys_open) rval=0x9\n[...]\n\nThat's a bit better.\n\n\nTracing the open() mode:\n\n# ./kprobe 'p:myopen do_sys_open mode=%cx:u16'\nTracing kprobe myopen. Ctrl-C to end.\n          kprobe-29572 [001] d... 6593503.353923: myopen: (do_sys_open+0x0/0x220) mode=0x1\n          kprobe-29572 [001] d... 6593503.353945: myopen: (do_sys_open+0x0/0x220) mode=0x0\n          kprobe-29572 [001] d... 6593503.354307: myopen: (do_sys_open+0x0/0x220) mode=0x5c00\n          kprobe-29572 [001] d... 6593503.354401: myopen: (do_sys_open+0x0/0x220) mode=0x0\n       supervise-1689  [000] d... 6593503.944125: myopen: (do_sys_open+0x0/0x220) mode=0x1a4\n       supervise-1688  [001] d... 6593503.944125: myopen: (do_sys_open+0x0/0x220) mode=0x1a4\n       supervise-1688  [001] d... 6593503.944606: myopen: (do_sys_open+0x0/0x220) mode=0x1a4\n       supervise-1689  [000] d... 6593503.944606: myopen: (do_sys_open+0x0/0x220) mode=0x1a4\n       supervise-1698  [000] d... 6593503.944728: myopen: (do_sys_open+0x0/0x220) mode=0x1a4\n       supervise-1698  [000] d... 6593503.945077: myopen: (do_sys_open+0x0/0x220) mode=0x1a4\n[...]\n\nHere I guessed that the mode was in register %cx, and cast it as a 16-bit\nunsigned integer (\":u16\"). Your platform and kernel may be different, and the\nmode may be in a different register. If fiddling with such registers becomes too\npainful or unreliable for you, consider installing kernel debuginfo and using\nthe named variables with perf_events \"perf probe\".\n\n\nTracing the open() filename:\n\n# ./kprobe 'p:myopen do_sys_open filename=+0(%si):string'\nTracing kprobe myopen. Ctrl-C to end.\n          kprobe-32369 [001] d... 6593706.999728: myopen: (do_sys_open+0x0/0x220) filename=\"/etc/ld.so.cache\"\n          kprobe-32369 [001] d... 6593706.999748: myopen: (do_sys_open+0x0/0x220) filename=\"/lib/x86_64-linux-gnu/libc.so.6\"\n          kprobe-32369 [001] d... 6593707.000092: myopen: (do_sys_open+0x0/0x220) filename=\"/usr/lib/locale/locale-archive\"\n          kprobe-32369 [001] d... 6593707.000176: myopen: (do_sys_open+0x0/0x220) filename=\"trace_pipe\"\n       supervise-1699  [000] d... 6593707.254970: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1689  [001] d... 6593707.254970: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1689  [001] d... 6593707.255432: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1699  [000] d... 6593707.255432: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1695  [001] d... 6593707.258805: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n[...]\n\nAs mentioned previously, the %si register may be different on your platform.\nIn this example, I cast it as a string.\n\n\nSpecifying a duration will buffer in-kernel (reducing overhead), and write at\nthe end. Here's tracing for 10 seconds, and writing to the \"out\" file:\n\n# ./kprobe -d 10 'p:myopen do_sys_open filename=+0(%si):string' > out\n\n\nYou can match on a single PID only:\n\n# ./kprobe -p 1696 'p:myopen do_sys_open filename=+0(%si):string' \nTracing kprobe myopen. Ctrl-C to end.\n       supervise-1696  [001] d... 6593773.677033: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1696  [001] d... 6593773.677332: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1696  [001] d... 6593774.697144: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1696  [001] d... 6593774.697675: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1696  [001] d... 6593775.717986: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n       supervise-1696  [001] d... 6593775.718499: myopen: (do_sys_open+0x0/0x220) filename=\"supervise/status.new\"\n^C\nEnding tracing...\n\nThis will only show events when that PID is on-CPU.\n\n\nThe -v option will show you the available variables you can use in custom\nfilters:\n\n# ./kprobe -v 'p:myopen do_sys_open filename=+0(%si):string' \nname: myopen\nID: 1443\nformat:\n\tfield:unsigned short common_type;\toffset:0;\tsize:2;\tsigned:0;\n\tfield:unsigned char common_flags;\toffset:2;\tsize:1;\tsigned:0;\n\tfield:unsigned char common_preempt_count;\toffset:3;\tsize:1;\tsigned:0;\n\tfield:int common_pid;\toffset:4;\tsize:4;\tsigned:1;\n\n\tfield:unsigned long __probe_ip;\toffset:8;\tsize:8;\tsigned:0;\n\tfield:__data_loc char[] filename;\toffset:16;\tsize:4;\tsigned:1;\n\nprint fmt: \"(%lx) filename=\\\"%s\\\"\", REC->__probe_ip, __get_str(filename)\n\n\nTracing filenames that end in \"stat\", by adding a filter:\n\n# ./kprobe 'p:myopen do_sys_open filename=+0(%si):string' 'filename ~ \"*stat\"'\nTracing kprobe myopen. Ctrl-C to end.\n        postgres-1172  [000] d... 6594028.787166: myopen: (do_sys_open+0x0/0x220) filename=\"pg_stat_tmp/pgstat.stat\"\n        postgres-1172  [001] d... 6594028.797410: myopen: (do_sys_open+0x0/0x220) filename=\"pg_stat_tmp/pgstat.stat\"\n        postgres-1172  [001] d... 6594028.797467: myopen: (do_sys_open+0x0/0x220) filename=\"pg_stat_tmp/pgstat.stat\"\n        postgres-4443  [001] d... 6594028.800908: myopen: (do_sys_open+0x0/0x220) filename=\"pg_stat_tmp/pgstat.stat\"\n        postgres-4443  [000] d... 6594028.811237: myopen: (do_sys_open+0x0/0x220) filename=\"pg_stat_tmp/pgstat.stat\"\n        postgres-4443  [000] d... 6594028.811290: myopen: (do_sys_open+0x0/0x220) filename=\"pg_stat_tmp/pgstat.stat\"\n^C\nEnding tracing...\n\nThis filtering is done in-kernel context.\n\n\nAs an example of tracing a deeper kernel function, lets trace bio_alloc() and\nentry registers:\n\n# ./kprobe 'p:myprobe bio_alloc %ax %bx %cx %dx'\nTracing kprobe myprobe. Ctrl-C to end.\n       supervise-3055  [000] 2172148.728250: myprobe: (bio_alloc+0x0/0x30) arg1=ffff880064acc8d0 arg2=ffff8800e56a7990 arg3=0 arg4=ffff880064acc910\n       supervise-3055  [000] 2172148.728527: myprobe: (bio_alloc+0x0/0x30) arg1=ffff880064acf948 arg2=ffff8800e56a7990 arg3=0 arg4=ffff880064acf988\n    jbd2/xvda1-8-212   [000] 2172149.749474: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800ad1f87b8 arg3=ffff8800ba22c06c arg4=8\n    jbd2/xvda1-8-212   [000] 2172149.749485: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d053a8 arg3=10f16c5bb arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749487: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d05958 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749488: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d05b60 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749489: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d05820 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749489: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d055b0 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749490: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff88006ff22ea0 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749491: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d1f000 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749492: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff880089d1f138 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749493: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff88005d267138 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749494: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff88005d267680 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.749495: myprobe: (bio_alloc+0x0/0x30) arg1=0 arg2=ffff88005d2675b0 arg3=5 arg4=0\n    jbd2/xvda1-8-212   [000] 2172149.751044: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800cc241ea0 arg3=445f0300 arg4=ffff8800effba000\n       supervise-3055  [000] 2172149.751095: myprobe: (bio_alloc+0x0/0x30) arg1=ffff880064acf948 arg2=ffff8800e56a7990 arg3=0 arg4=ffff880064acf988\n       supervise-3055  [000] 2172149.751341: myprobe: (bio_alloc+0x0/0x30) arg1=ffff880064acc8d0 arg2=ffff8800e56a7990 arg3=0 arg4=ffff880064acc910\n       supervise-3055  [000] 2172150.772033: myprobe: (bio_alloc+0x0/0x30) arg1=ffff880064acc8d0 arg2=ffff8800e56a7990 arg3=0 arg4=ffff880064acc910\n       supervise-3055  [000] 2172150.772305: myprobe: (bio_alloc+0x0/0x30) arg1=ffff880064acf948 arg2=ffff8800e56a7990 arg3=0 arg4=ffff880064acf988\n     flush-202:1-409   [000] 2172151.087815: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800da51d6e8 arg3=16afd arg4=1\n     flush-202:1-409   [000] 2172151.087829: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800e7537f08 arg3=16afd arg4=2\n     flush-202:1-409   [000] 2172151.087844: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800e7519af8 arg3=16afd arg4=3\n     flush-202:1-409   [000] 2172151.087846: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800e7511478 arg3=16afd arg4=4\n     flush-202:1-409   [000] 2172151.087849: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800e75e6a90 arg3=16afd arg4=5\n     flush-202:1-409   [000] 2172151.087851: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800e7512bc8 arg3=16afd arg4=6\n     flush-202:1-409   [000] 2172151.087853: myprobe: (bio_alloc+0x0/0x30) arg1=ffffffff arg2=ffff8800eb3bf410 arg3=16afd arg4=7\n^C\n\nThe output includes who is on-CPU, high resolution timestamps, and the arguments\nwe requested (registers %ax to %dx). These registers are platform dependent,\nand are mapped by the compiler to the entry arguments of the function.\n\nHow are these useful? If you are debugging this kernel function, you'll know. :)\n\n\nNote that you can add qualifiers, eg, if I knew %ax was a uint32:\n\n# ./kprobe 'p:myprobe bio_alloc %ax:u32'\nTracing kprobe myprobe. Ctrl-C to end.\n       supervise-3055  [000] 2172389.734606: myprobe: (bio_alloc+0x0/0x30) arg1=64acf948\n       supervise-3055  [000] 2172389.734865: myprobe: (bio_alloc+0x0/0x30) arg1=64acc8d0\n       supervise-3055  [000] 2172390.772391: myprobe: (bio_alloc+0x0/0x30) arg1=64acf948\n       supervise-3055  [000] 2172390.772676: myprobe: (bio_alloc+0x0/0x30) arg1=64acc8d0\n^C\nEnding tracing...\n\nYou can give them aliases too, instead of the default arg1..N:\n\n# ./kprobe 'p:myprobe bio_alloc ax=%ax'\nTracing kprobe myprobe. Ctrl-C to end.\n       supervise-3055  [000] 2172420.451663: myprobe: (bio_alloc+0x0/0x30) ax=ffff880064acc8d0\n       supervise-3055  [000] 2172420.451938: myprobe: (bio_alloc+0x0/0x30) ax=ffff880064acf948\n     flush-202:1-409   [000] 2172421.163462: myprobe: (bio_alloc+0x0/0x30) ax=ffff880064acc8d0\n       supervise-3055  [000] 2172421.500994: myprobe: (bio_alloc+0x0/0x30) ax=ffff880064acc8d0\n       supervise-3055  [000] 2172421.501307: myprobe: (bio_alloc+0x0/0x30) ax=ffff880064acf948\n^C\nEnding tracing...\n\n\nNow for the return of bio_alloc():\n\n# ./kprobe 'r:myprobe bio_alloc $retval'\nTracing kprobe myprobe. Ctrl-C to end.\n       supervise-3055  [000] 2172164.145533: myprobe: (io_submit_init.isra.6+0x74/0x100 <- bio_alloc) arg1=ffff8800e55843c0\n       supervise-3055  [000] 2172164.145829: myprobe: (io_submit_init.isra.6+0x74/0x100 <- bio_alloc) arg1=ffff8800e5584840\n    jbd2/xvda1-8-212   [000] 2172165.166453: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e57596c0\n    jbd2/xvda1-8-212   [000] 2172165.166493: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759c00\n    jbd2/xvda1-8-212   [000] 2172165.166496: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759600\n    jbd2/xvda1-8-212   [000] 2172165.166497: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759e40\n    jbd2/xvda1-8-212   [000] 2172165.166498: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e57590c0\n    jbd2/xvda1-8-212   [000] 2172165.166500: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e57599c0\n    jbd2/xvda1-8-212   [000] 2172165.166500: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759a80\n    jbd2/xvda1-8-212   [000] 2172165.166502: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759f00\n    jbd2/xvda1-8-212   [000] 2172165.166503: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759540\n    jbd2/xvda1-8-212   [000] 2172165.166504: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759180\n    jbd2/xvda1-8-212   [000] 2172165.166504: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759900\n    jbd2/xvda1-8-212   [000] 2172165.166505: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759000\n    jbd2/xvda1-8-212   [000] 2172165.166506: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759480\n           <...>-212   [000] 2172165.176261: myprobe: (submit_bh+0x76/0x120 <- bio_alloc) arg1=ffff8800e5759480\n       supervise-3055  [000] 2172165.176317: myprobe: (io_submit_init.isra.6+0x74/0x100 <- bio_alloc) arg1=ffff8800e57596c0\n       supervise-3055  [000] 2172165.176586: myprobe: (io_submit_init.isra.6+0x74/0x100 <- bio_alloc) arg1=ffff8800e5759900\n^C\nEnding tracing...\n\nGreat. This output includes the function we are returning to, in most cases,\nsubmit_bh().\n\nNote that this mode (without a duration) prints events as they happen,\nso the overheads can be high for frequent events. You could try the -d mode,\nwhich buffers in-kernel.\n\n\nThe -s option will print the kernel stack trace after the event:\n\n# ./kprobe -s 'p:mytcp tcp_init_cwnd'\nTracing kprobe mytcp. Ctrl-C to end.\n            sshd-5121  [000] d... 6897275.911301: mytcp: (tcp_init_cwnd+0x0/0x40)\n            sshd-5121  [000] d... 6897275.911309: <stack trace>\n => tcp_write_xmit\n => __tcp_push_pending_frames\n => tcp_push\n => tcp_sendmsg\n => inet_sendmsg\n => sock_aio_write\n => do_sync_write\n => vfs_write\n => SyS_write\n => system_call_fastpath\n            sshd-32219 [000] d... 6897275.911467: mytcp: (tcp_init_cwnd+0x0/0x40)\n            sshd-32219 [000] d... 6897275.911471: <stack trace>\n => tcp_write_xmit\n => __tcp_push_pending_frames\n => tcp_push\n => tcp_sendmsg\n => inet_sendmsg\n => sock_aio_write\n => do_sync_write\n => vfs_write\n => SyS_write\n => system_call_fastpath\n            sshd-5121  [000] d... 6897277.878794: mytcp: (tcp_init_cwnd+0x0/0x40)\n            sshd-5121  [000] d... 6897277.878801: <stack trace>\n => tcp_write_xmit\n => __tcp_push_pending_frames\n => tcp_push\n => tcp_sendmsg\n => inet_sendmsg\n => sock_aio_write\n => do_sync_write\n => vfs_write\n => SyS_write\n => system_call_fastpath\n\nThis makes use of the kernel options/stacktrace feature.\n\n\nUse -h to print the USAGE message:\n\n# ./kprobe -h\nUSAGE: kprobe [-FhHsv] [-d secs] [-p PID] [-L TID] kprobe_definition [filter]\n                 -F              # force. trace despite warnings.\n                 -d seconds      # trace duration, and use buffers\n                 -p PID          # PID to match on events\n                 -L TID          # thread id to match on events\n                 -v              # view format file (don't trace)\n                 -H              # include column headers\n                 -s              # show kernel stack traces\n                 -h              # this usage message\n\nNote that these examples may need modification to match your kernel\nversion's function names and platform's register usage.\n   eg,\n       kprobe p:do_sys_open\n                                 # trace open() entry\n       kprobe r:do_sys_open\n                                 # trace open() return\n       kprobe 'r:do_sys_open $retval'\n                                 # trace open() return value\n       kprobe 'r:myopen do_sys_open $retval'\n                                 # use a custom probe name\n       kprobe 'p:myopen do_sys_open mode=%cx:u16'\n                                 # trace open() file mode\n       kprobe 'p:myopen do_sys_open filename=+0(%si):string'\n                                 # trace open() with filename\n       kprobe -s 'p:myprobe tcp_retransmit_skb'\n                                 # show kernel stacks\n       kprobe 'p:do_sys_open file=+0(%si):string' 'file ~ \"*stat\"'\n                                 # opened files ending in \"stat\"\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/opensnoop_example.txt",
    "content": "Demonstrations of opensnoop, the Linux ftrace version.\n\n\n# ./opensnoop\nTracing open()s. Ctrl-C to end.\nCOMM             PID      FD FILE\nopensnoop        5334    0x3 \n<...>            5343    0x3 /etc/ld.so.cache\nopensnoop        5342    0x3 /etc/ld.so.cache\n<...>            5343    0x3 /lib/x86_64-linux-gnu/libc.so.6\nopensnoop        5342    0x3 /lib/x86_64-linux-gnu/libm.so.6\nopensnoop        5342    0x3 /lib/x86_64-linux-gnu/libc.so.6\n<...>            5343    0x3 /usr/lib/locale/locale-archive\n<...>            5343    0x3 trace_pipe\nsupervise        1684    0x9 supervise/status.new\nsupervise        1684    0x9 supervise/status.new\nsupervise        1688    0x9 supervise/status.new\nsupervise        1688    0x9 supervise/status.new\nsupervise        1686    0x9 supervise/status.new\nsupervise        1685    0x9 supervise/status.new\nsupervise        1685    0x9 supervise/status.new\nsupervise        1686    0x9 supervise/status.new\n[...]\n\nThe first several lines show opensnoop catching itself initializing.\n\n\nUse -h to print the USAGE message:\n\n# ./opensnoop -h\nUSAGE: opensnoop [-htx] [-d secs] [-p PID] [-L TID] [-n name] [filename]\n                 -d seconds      # trace duration, and use buffers\n                 -n name         # process name to match on open\n                 -p PID          # PID to match on open\n                 -L TID          # thread id to match on open\n                 -t              # include time (seconds)\n                 -x              # only show failed opens\n                 -h              # this usage message\n                 filename        # match filename (partials, REs, ok)\n  eg,\n       opensnoop                 # watch open()s live (unbuffered)\n       opensnoop -d 1            # trace 1 sec (buffered)\n       opensnoop -p 181          # trace I/O issued by PID 181 only\n       opensnoop conf            # trace filenames containing \"conf\"\n       opensnoop 'log$'          # filenames ending in \"log\"\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/perf-stat-hist_example.txt",
    "content": "Demonstrations of perf-stat-hist, the Linux perf_events version.\n\n\nTracing the net:net_dev_xmit tracepoint, and building a power-of-4 histogram\nfor the \"len\" variable, for 10 seconds:\n\n# ./perf-stat-hist net:net_dev_xmit len 10\nTracing net:net_dev_xmit, power-of-4, max 1048576, for 10 seconds...\n\n            Range          : Count    Distribution\n            0              : 0        |                                      |\n            1 -> 3         : 0        |                                      |\n            4 -> 15        : 0        |                                      |\n           16 -> 63        : 2        |#                                     |\n           64 -> 255       : 30       |###                                   |\n          256 -> 1023      : 3        |#                                     |\n         1024 -> 4095      : 446      |######################################|\n         4096 -> 16383     : 0        |                                      |\n        16384 -> 65535     : 0        |                                      |\n        65536 -> 262143    : 0        |                                      |\n       262144 -> 1048575   : 0        |                                      |\n      1048576 ->           : 0        |                                      |\n\nThis showed that most of the network transmits were between 1024 and 4095 bytes,\nwith a handful between 64 and 255 bytes.\n\nCat the format file for the tracepoint to see what other variables are available\nto trace. Eg:\n\n# cat /sys/kernel/debug/tracing/events/net/net_dev_xmit/format \nname: net_dev_xmit\nID: 1078\nformat:\n\tfield:unsigned short common_type;\toffset:0;\tsize:2;\tsigned:0;\n\tfield:unsigned char common_flags;\toffset:2;\tsize:1;\tsigned:0;\n\tfield:unsigned char common_preempt_count;\toffset:3;\tsize:1;\tsigned:0;\n\tfield:int common_pid;\toffset:4;\tsize:4;\tsigned:1;\n\n\tfield:void * skbaddr;\toffset:8;\tsize:8;\tsigned:0;\n\tfield:unsigned int len;\toffset:16;\tsize:4;\tsigned:0;\n\tfield:int rc;\toffset:20;\tsize:4;\tsigned:1;\n\tfield:__data_loc char[] name;\toffset:24;\tsize:4;\tsigned:1;\n\nprint fmt: \"dev=%s skbaddr=%p len=%u rc=%d\", __get_str(name), REC->skbaddr, REC->len, REC->rc\n\nThat's where \"len\" came from.\n\nThis works by creating a series of tracepoint and filter pairs for each\nhistogram bucket, and doing in-kernel counts. The overhead should in many cases\nbe better than user space post-processing, however, this approach is still\nnot ideal. I've called it a \"perf hacktogram\". The overhead is relative to\nthe frequency of events, multiplied by the number of buckets. You can modify\nthe script to use power-of-2 instead, or whatever you like, but the overhead\nfor more buckets will be higher.\n\n\nHistogram of the returned read() syscall sizes:\n\n# ./perf-stat-hist syscalls:sys_exit_read ret 10\nTracing syscalls:sys_exit_read, power-of-4, max 1048576, for 10 seconds...\n\n            Range          : Count    Distribution\n            0              : 90       |#                                     |\n            1 -> 3         : 9587     |######################################|\n            4 -> 15        : 69       |#                                     |\n           16 -> 63        : 590      |###                                   |\n           64 -> 255       : 250      |#                                     |\n          256 -> 1023      : 389      |##                                    |\n         1024 -> 4095      : 296      |##                                    |\n         4096 -> 16383     : 183      |#                                     |\n        16384 -> 65535     : 12       |#                                     |\n        65536 -> 262143    : 0        |                                      |\n       262144 -> 1048575   : 0        |                                      |\n      1048576 ->           : 0        |                                      |\n\nMost of our read()s were tiny, between 1 and 3 bytes.\n\n\nUsing power-of-2, and a max of 1024:\n\n# ./perf-stat-hist -P 2 -m 1024 syscalls:sys_exit_read ret\nTracing syscalls:sys_exit_read, power-of-2, max 1024, until Ctrl-C...\n^C\n            Range          : Count    Distribution\n              -> -1        : 29       |##                                    |\n            0 -> 0         : 1        |#                                     |\n            1 -> 1         : 959      |######################################|\n            2 -> 3         : 1        |#                                     |\n            4 -> 7         : 0        |                                      |\n            8 -> 15        : 2        |#                                     |\n           16 -> 31        : 14       |#                                     |\n           32 -> 63        : 1        |#                                     |\n           64 -> 127       : 0        |                                      |\n          128 -> 255       : 0        |                                      |\n          256 -> 511       : 0        |                                      |\n          512 -> 1023      : 1        |#                                     |\n         1024 ->           : 1        |#                                     |\n\n\nSpecifying custom bucket sizes:\n\n# ./perf-stat-hist -b \"10 50 100 5000\" syscalls:sys_exit_read ret\nTracing syscalls:sys_exit_read, specified buckets, until Ctrl-C...\n^C\n            Range          : Count    Distribution\n              -> 9         : 989      |######################################|\n           10 -> 49        : 5        |#                                     |\n           50 -> 99        : 0        |                                      |\n          100 -> 4999      : 2        |#                                     |\n         5000 ->           : 0        |                                      |\n\n\nSpecifying a single value to bifurcate statistics:\n\n# ./perf-stat-hist -b 10 syscalls:sys_exit_read ret\nTracing syscalls:sys_exit_read, specified buckets, until Ctrl-C...\n^C\n            Range          : Count    Distribution\n              -> 9         : 2959     |######################################|\n           10 ->           : 7        |#                                     |\n\nThis has the lowest overhead for collection, since only two tracepoint \nfilter pairs are used.\n\n\nUse -h to print the USAGE message:\n\n# ./perf-stat-hist -h\nUSAGE: perf-stat-hist [-h] [-b buckets|-P power] [-m max] tracepoint\n                      variable [seconds]\n                 -b buckets      # specify histogram bucket points\n                 -P power        # power-of (default is 4)\n                 -m max          # max value for power-of\n                 -h              # this usage message\n   eg,\n       perf-stat-hist syscalls:sys_enter_read count 5\n                 # read() request histogram, 5 seconds\n       perf-stat-hist syscalls:sys_exit_read ret 5\n                 # read() return histogram, 5 seconds\n       perf-stat-hist -P 10 syscalls:sys_exit_read ret 5\n                 # ... use power-of-10\n       perf-stat-hist -P 2 -m 1024 syscalls:sys_exit_read ret 5\n                 # ... use power-of-2, max 1024\n       perf-stat-hist -b \"10 50 100 500\" syscalls:sys_exit_read ret 5\n                 # ... histogram based on these bucket ranges\n       perf-stat-hist -b 10 syscalls:sys_exit_read ret 5\n                 # ... bifurcate by the value 10 (lowest overhead)\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/reset-ftrace_example.txt",
    "content": "Demonstrations of reset-ftrace, the Linux ftrace tool.\n\n\nYou will probably never need this tool. If you kill -9 an ftrace-based tool,\nleaving the kernel in a tracing enabled state, you could try using this tool\nto reset ftrace and disable tracing. Make sure no other ftrace sessions are\nin use on your system, or it will kill those.\n\nHere's an example:\n\n# ./opensnoop\nTracing open()s. Ctrl-C to end.\nERROR: ftrace may be in use by PID 2197 /var/tmp/.ftrace-lock\n\nI tried to run opensnoop, but there's a lock file for PID 2197. Checking if it\nexists:\n\n# ps -fp 2197\nUID        PID  PPID  C STIME TTY          TIME CMD\n#\n\nNo.\n\nI also know that no one is using ftrace on this system. So I'll use reset-ftrace\nto clean up this lock file and ftrace state:\n\n# ./reset-ftrace \nERROR: ftrace lock (/var/tmp/.ftrace-lock) exists. It shows ftrace may be in use by PID 2197.\nDouble check to see if that PID is still active. If not, consider using -f to force a reset. Exiting.\n\n... except it's complaining about the lock file too. I'm already sure that this\nPID doesn't exist, so I'll add the -f option:\n\n# ./reset-ftrace -f\nReseting ftrace state...\n\ncurrent_tracer, before:\n     1\tnop\ncurrent_tracer, after:\n     1\tnop\n\nset_ftrace_filter, before:\n     1\t#### all functions enabled ####\nset_ftrace_filter, after:\n     1\t#### all functions enabled ####\n\nset_ftrace_pid, before:\n     1\tno pid\nset_ftrace_pid, after:\n     1\tno pid\n\nkprobe_events, before:\nkprobe_events, after:\n\nDone.\n\nThe output shows what has been reset, including the before and after state of\nthese files.\n\nNow I can try iosnoop again:\n\n# ./iosnoop\nTracing block I/O. Ctrl-C to end.\nCOMM             PID    TYPE DEV      BLOCK        BYTES     LATms\nsupervise        1689   W    202,1    17039664     4096       0.58\nsupervise        1689   W    202,1    17039672     4096       0.47\nsupervise        1694   W    202,1    17039744     4096       0.98\nsupervise        1694   W    202,1    17039752     4096       0.74\nsupervise        1684   W    202,1    17039760     4096       0.63\n[...]\n\nFixed.\n\nNote that reset-ftrace currently only resets a few methods of enabling\ntracing, such as set_ftrace_filter and kprobe_events. Static tracepoints could \nbe enabled individually, and this script currently doesn't find and disable\nthose.\n\n\nUse -h to print the USAGE message:\n\n# ./reset-ftrace -h\nUSAGE: reset-ftrace [-fhq]\n                 -f              # force: delete ftrace lock file\n                 -q              # quiet: reset, but say nothing\n                 -h              # this usage message\n  eg,\n       reset-ftrace              # disable active ftrace session\n"
  },
  {
    "path": "examples/syscount_example.txt",
    "content": "Demonstrations of syscount, the Linux perf_events version.\n\n\nThe first mode I use is \"-c\", where it behaves like \"strace -c\", but for the\nentire system (all procesess) and with much lower overhead:\n\n# ./syscount -c\nTracing... Ctrl-C to end.\n^Csleep: Interrupt\nSYSCALL              COUNT\naccept                   1\ngetsockopt               1\nsetsid                   1\nchdir                    2\ngetcwd                   2\ngetpeername              2\ngetsockname              2\nsetgid                   2\nsetgroups                2\nsetpgid                  2\nsetuid                   2\ngetpgrp                  4\ngetpid                   4\nrename                   4\nsetitimer                4\nsetrlimit                4\nsetsockopt               4\nstatfs                   4\nset_tid_address          5\nreadlink                 6\nset_robust_list          6\nnanosleep                7\nnewuname                 7\nfaccessat                8\nfutex                   10\nclock_gettime           16\nnewlstat                20\npipe                    20\nepoll_wait              24\ngetrlimit               25\nsocket                  27\nconnect                 29\nexit_group              30\ngetppid                 31\ndup2                    34\nwait4                   51\nfcntl                   58\ngetegid                 72\ngetgid                  72\ngetuid                  72\ngeteuid                 75\nperf_event_open        100\nmunmap                 121\ngettimeofday           216\naccess                 266\nioctl                  340\npoll                   348\nsendto                 374\nmprotect               414\nbrk                    597\nrt_sigaction           632\nrecvfrom               664\nlseek                  749\nnewfstatat            2922\nopenat                2925\nnewfstat              3229\nnewstat               4334\nopen                  4534\nfchdir                5845\ngetdents              5854\nread                  7673\nclose                 7728\nselect                9633\nrt_sigprocmask       19886\nwrite                34581\n\nWhile tracing, the write() syscall was executed 34,581 times.\n\nThis mode uses \"perf stat\" to count the syscalls:* tracepoints in-kernel.\n\n\nYou can add a duration (-d) and limit the number shown (-t):\n\n# ./syscount -cd 5 -t 10\nTracing for 5 seconds. Top 10 only...\nSYSCALL              COUNT\ngettimeofday          1009\nwrite                 3583\nread                  8174\nopenat               21550\nnewfstat             21558\nopen                 21824\nfchdir               43098\ngetdents             43106\nclose                43694\nnewfstatat          110936\n\nWhile tracing for 5 seconds, the newfstatat() syscall was executed 110,936\ntimes.\n\n\nWithout the -c, syscount shows syscalls by process name:\n\n# ./syscount -d 5 -t 10\nTracing for 5 seconds. Top 10 only...\n[ perf record: Woken up 66 times to write data ]\n[ perf record: Captured and wrote 16.513 MB perf.data (~721455 samples) ]\nCOMM                COUNT\nstat                  450\nperl                  537\ncatalina.sh          1700\npostgres             2094\nrun                  2362\n:6946                4764\nps                   5961\nsshd                45796\nfind                61039\n\nSo processes named \"find\" called 61,039 syscalls during the 5 seconds of\ntracing.\n\nNote that this mode writes a perf.data file. This is higher overhead for a\nfew reasons:\n\n- all data is passed from kernel to user space, which eats CPU for the memory\n  copy. Note that it is buffered in an efficient way by perf_events, which\n  wakes up and context switches only a small number of times: 66 in this case,\n  to hand 16 Mbytes of trace data to user space.\n- data is post-processed in user space, eating more CPU.\n- data is stored on the file system in the perf.data file, consuming available\n  storage.\n\nThis will be improved in future kernels, but it is difficult to improve this\nmuch further in existing kernels. For example, using a pipe to \"perf script\"\ninstead of writing perf.data can have issues with feedback loops, where\nperf traces itself. This syscount version goes to lengths to avoid tracing\nits own perf, but \nright now with existing functionality in older kernels. The trip via perf.data\nis necessary\n\n\nRunning without options shows syscalls by process name until Ctrl-C:\n\n# ./syscount \nTracing... Ctrl-C to end.\n^C[ perf record: Woken up 39 times to write data ]\n[ perf record: Captured and wrote 9.644 MB perf.data (~421335 samples) ]\nCOMM                COUNT\napache2                 8\napacheLogParser        13\nplatformservice        16\nsnmpd                  16\nntpd                   21\nmultilog               66\nsupervise              84\ndirname               102\necho                  102\nsvstat                108\ncut                   111\nbash                  113\ngrep                  132\nxargs                 132\nredis-server          190\nsed                   192\nsetuidgid             294\nstat                  450\nperl                  537\ncatalina.sh          1275\npostgres             1736\nrun                  2352\n:7396                4527\nps                   5925\nsshd                20154\nfind                28700\n\nNote again it is writing a perf.data file to do this.\n\n\nThe -v option adds process IDs:\n\n# ./syscount -v\nTracing... Ctrl-C to end.\n^C[ perf record: Woken up 48 times to write data ]\n[ perf record: Captured and wrote 12.114 MB perf.data (~529276 samples) ]\nPID    COMM                COUNT\n3599   apacheLogParser         3\n7977   xargs                   3\n7982   supervise               3\n7993   xargs                   3\n3575   apache2                 4\n1311   ntpd                    6\n3135   postgres                6\n3600   apacheLogParser         6\n3210   platformservice         8\n6503   sshd                    9\n7978   :7978                   9\n7994   run                     9\n7968   :7968                  11\n7984   run                    11\n1451   snmpd                  16\n3040   svscan                 17\n3066   postgres               17\n3133   postgres               24\n3134   postgres               24\n3136   postgres               24\n3061   multilog               29\n3055   supervise              30\n7979   bash                   31\n7977   echo                   34\n7981   dirname                34\n7993   echo                   34\n7968   svstat                 36\n7984   svstat                 36\n7975   cut                    37\n7991   cut                    37\n9857   bash                   37\n7967   :7967                  40\n7983   run                    40\n7972   :7972                  41\n7976   xargs                  41\n7988   run                    41\n7992   xargs                  41\n7969   :7969                  42\n7976   :7976                  42\n7985   run                    42\n7992   run                    42\n7973   :7973                  43\n7974   :7974                  43\n7989   run                    43\n7990   run                    43\n7973   grep                   44\n7989   grep                   44\n7975   :7975                  45\n7991   run                    45\n7970   :7970                  51\n7986   run                    51\n7981   catalina.sh            52\n7974   sed                    64\n7990   sed                    64\n3455   postgres               66\n7971   :7971                  66\n7987   run                    66\n7966   :7966                  96\n7966   setuidgid              98\n3064   redis-server          110\n7970   stat                  150\n7986   stat                  150\n7969   perl                  179\n7985   perl                  179\n7982   run                   341\n7966   catalina.sh           373\n7980   postgres              432\n7972   ps                   1971\n7988   ps                   1983\n9832   sshd                37511\n7979   find                51040\n\nOnce you've found a process ID of interest, you can use \"-c\" and \"-p PID\" to\nshow syscall names. This also switches to \"perf stat\" mode for in-kernel\ncounts, and lower overhead:\n\n# ./syscount -cp 7979\nTracing PID 7979... Ctrl-C to end.\n^CSYSCALL              COUNT\nbrk                     10\nnewfstat              2171\nopen                  2171\nnewfstatat            2175\nopenat                2175\nclose                 4346\nfchdir                4346\ngetdents              4351\nwrite                25482\n\nSo the most frequent syscall by PID 7979 was write().\n\n\nUse -h to print the USAGE message:\n\n# ./syscount -h\nUSAGE: syscount [-chv] [-t top] {-p PID|-d seconds|command}\n       syscount                  # count by process name\n                -c               # show counts by syscall name\n                -h               # this usage message\n                -v               # verbose: shows PID\n                -p PID           # trace this PID only\n                -d seconds       # duration of trace\n                -t num           # show top number only\n                command          # run and trace this command\n  eg,\n        syscount                 # syscalls by process name\n        syscount -c              # syscalls by syscall name\n        syscount -d 5            # trace for 5 seconds\n        syscount -cp 923         # syscall names for PID 923\n        syscount -c ls           # syscall names for \"ls\"\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/tcpretrans_example.txt",
    "content": "Demonstrations of tcpretrans, the Linux ftrace version.\n\n\nTracing TCP retransmits on a busy server:\n\n# ./tcpretrans \nTIME     PID    LADDR:LPORT          -- RADDR:RPORT          STATE\n05:16:44 3375   10.150.18.225:53874  R> 10.105.152.3:6001    ESTABLISHED\n05:16:44 3375   10.150.18.225:53874  R> 10.105.152.3:6001    ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED\n05:16:55 0      10.150.18.225:47115  R> 10.71.171.158:6001   ESTABLISHED\n05:16:58 0      10.150.18.225:44388  R> 10.103.130.120:6001  ESTABLISHED\n05:16:58 0      10.150.18.225:44388  R> 10.103.130.120:6001  ESTABLISHED\n05:16:58 0      10.150.18.225:44388  R> 10.103.130.120:6001  ESTABLISHED\n05:16:59 0      10.150.18.225:56086  R> 10.150.32.107:6001   ESTABLISHED\n05:16:59 0      10.150.18.225:56086  R> 10.150.32.107:6001   ESTABLISHED\n^C\nEnding tracing...\n\nThis shows TCP retransmits by dynamically tracing the kernel function that does\nthe retransmit. This is a low overhead approach.\n\nThe PID may or may not make sense: it's showing the PID that was on-CPU,\nhowever, retransmits are often timer-based, where it's the kernel that is\non-CPU.\n\nThe STATE column shows the TCP state for the socket performing the retransmit.\nThe \"--\" column is the packet type. \"R>\" for retransmit.\n\n\nKernel stack traces can be included with -s, which may show the type of\nretransmit:\n\n# ./tcpretrans -s\nTIME     PID    LADDR:LPORT          -- RADDR:RPORT          STATE\n06:21:10 19516  10.144.107.151:22    R> 10.13.106.251:32167  ESTABLISHED\n => tcp_fastretrans_alert\n => tcp_ack\n => tcp_rcv_established\n => tcp_v4_do_rcv\n => tcp_v4_rcv\n => ip_local_deliver_finish\n => ip_local_deliver\n => ip_rcv_finish\n => ip_rcv\n => __netif_receive_skb\n => netif_receive_skb\n => handle_incoming_queue\n => xennet_poll\n => net_rx_action\n => __do_softirq\n => call_softirq\n => do_softirq\n => irq_exit\n => xen_evtchn_do_upcall\n => xen_do_hypervisor_callback\n\nThis looks like a fast retransmit (inclusion of tcp_fastretrans_alert(), and\nbeing based on receiving an ACK, rather than a timer).\n\n\nThe -l option will include TCP tail loss probe events (TLP; see\nhttp://lwn.net/Articles/542642/). Eg:\n\n# ./tcpretrans -l\nTIME     PID    LADDR:LPORT          -- RADDR:RPORT          STATE       \n21:56:06 0      10.100.155.200:22    R> 10.10.237.72:18554   LAST_ACK    \n21:56:08 0      10.100.155.200:22    R> 10.10.237.72:18554   LAST_ACK    \n21:56:10 16452  10.100.155.200:22    R> 10.10.237.72:18554   LAST_ACK    \n21:56:10 0      10.100.155.200:22    L> 10.10.237.72:46408   LAST_ACK    \n21:56:10 0      10.100.155.200:22    R> 10.10.237.72:46408   LAST_ACK    \n21:56:12 0      10.100.155.200:22    R> 10.10.237.72:46408   LAST_ACK    \n21:56:13 0      10.100.155.200:22    R> 10.10.237.72:46408   LAST_ACK    \n^C\nEnding tracing...\n\nLook for \"L>\" in the type column (\"--\") for TLP events.\n\n\nUse -h to print the USAGE message:\n\n# ./tcpretrans -h\nUSAGE: tcpretrans [-hs]\n                  -h       # help message\n                  -s       # print stack traces\n   eg,\n       tcpretrans          # trace TCP retransmits\n"
  },
  {
    "path": "examples/tpoint_example.txt",
    "content": "Demonstrations of tpoint, the Linux ftrace version.\n\n\nLet's trace block:block_rq_issue, to see block device (disk) I/O requests:\n\n# ./tpoint block:block_rq_issue \nTracing block:block_rq_issue. Ctrl-C to end.\n       supervise-1692  [001] d... 7269912.982162: block_rq_issue: 202,1 W 0 () 17039656 + 8 [supervise]\n       supervise-1696  [000] d... 7269912.982243: block_rq_issue: 202,1 W 0 () 12862264 + 8 [supervise]\n           cksum-12994 [000] d... 7269913.317924: block_rq_issue: 202,1 R 0 () 9357056 + 72 [cksum]\n           cksum-12994 [000] d... 7269913.319013: block_rq_issue: 202,1 R 0 () 2977536 + 144 [cksum]\n           cksum-12994 [000] d... 7269913.320217: block_rq_issue: 202,1 R 0 () 2986240 + 216 [cksum]\n           cksum-12994 [000] d... 7269913.321677: block_rq_issue: 202,1 R 0 () 620344 + 56 [cksum]\n           cksum-12994 [001] d... 7269913.329309: block_rq_issue: 202,1 R 0 () 9107912 + 88 [cksum]\n           cksum-12994 [001] d... 7269913.340133: block_rq_issue: 202,1 R 0 () 3147008 + 248 [cksum]\n           cksum-12994 [001] d... 7269913.354551: block_rq_issue: 202,1 R 0 () 11583488 + 256 [cksum]\n           cksum-12994 [001] d... 7269913.379904: block_rq_issue: 202,1 R 0 () 11583744 + 256 [cksum]\n[...]\n^C\nEnding tracing...\n\nGreat, that was easy!\n\nperf_events can do this as well, and is better in many ways, including a more\nefficient buffering strategy, and multi-user access. It's not that easy to do\nthis one-liner in perf_events, however. An equivalent for recent kernels is:\n\nperf record --no-buffer -e block:block_rq_issue -a -o - | PAGER=cat stdbuf -oL perf script -i -\n\nOlder kernels, use -D instead of --no-buffer. Even better is to set the buffer\npage size to a sufficient grouping (using -m), to minimize overheads, at the\nexpense of liveliness of updates. Note that stack traces (-g) don't work on\nmy systems with this perf one-liner, however, they do work with tpoint -s.\n\n\nColumn headings can be printed using -H:\n\n# ./tpoint -H block:block_rq_issue \nTracing block:block_rq_issue. Ctrl-C to end.\n# tracer: nop\n#\n# entries-in-buffer/entries-written: 0/0   #P:2\n#\n#                              _-----=> irqs-off\n#                             / _----=> need-resched\n#                            | / _---=> hardirq/softirq\n#                            || / _--=> preempt-depth\n#                            ||| /     delay\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#              | |       |   ||||       |         |\n       supervise-1697  [000] d... 7270545.340856: block_rq_issue: 202,1 W 0 () 12862464 + 8 [supervise]\n       supervise-1697  [000] d... 7270545.341256: block_rq_issue: 202,1 W 0 () 12862472 + 8 [supervise]\n       supervise-1690  [000] d... 7270545.342363: block_rq_issue: 202,1 W 0 () 17040368 + 8 [supervise]\n[...]\n\nThey are also documented in the Linux kernel source under:\nDocumentation/trace/ftrace.txt.\n\n\nHow about stacks traces for those block_rq_issue events? Adding -s:\n\n# ./tpoint -s block:block_rq_issue\nTracing block:block_rq_issue. Ctrl-C to end.\n       supervise-1691  [000] d... 7269511.079179: block_rq_issue: 202,1 W 0 () 17040232 + 8 [supervise]\n       supervise-1691  [000] d... 7269511.079188: <stack trace>\n => blk_peek_request\n => do_blkif_request\n => __blk_run_queue\n => queue_unplugged\n => blk_flush_plug_list\n => blk_finish_plug\n => ext4_writepages\n => do_writepages\n => __filemap_fdatawrite_range\n => filemap_flush\n => ext4_alloc_da_blocks\n => ext4_rename\n => vfs_rename\n => SYSC_renameat2\n => SyS_renameat2\n => SyS_rename\n => system_call_fastpath\n           cksum-7428  [000] d... 7269511.331778: block_rq_issue: 202,1 R 0 () 9006848 + 208 [cksum]\n           cksum-7428  [000] d... 7269511.331784: <stack trace>\n => blk_peek_request\n => do_blkif_request\n => __blk_run_queue\n => queue_unplugged\n => blk_flush_plug_list\n => blk_finish_plug\n => __do_page_cache_readahead\n => ondemand_readahead\n => page_cache_async_readahead\n => generic_file_read_iter\n => new_sync_read\n => vfs_read\n => SyS_read\n => system_call_fastpath\n           cksum-7428  [000] d... 7269511.332631: block_rq_issue: 202,1 R 0 () 620992 + 200 [cksum]\n           cksum-7428  [000] d... 7269511.332639: <stack trace>\n => blk_peek_request\n => do_blkif_request\n => __blk_run_queue\n => queue_unplugged\n => blk_flush_plug_list\n => blk_finish_plug\n => __do_page_cache_readahead\n => ondemand_readahead\n => page_cache_sync_readahead\n => generic_file_read_iter\n => new_sync_read\n => vfs_read\n => SyS_read\n => system_call_fastpath\n^C\nEnding tracing...\n\nEasy. Now I can read the ancestry to understand what actually lead to issuing\na block device (disk) I/O.\n\n\nHere's insertion onto the block I/O queue (better matches processes):\n\n# ./tpoint -s block:block_rq_insert\nTracing block:block_rq_insert. Ctrl-C to end.\n           cksum-11908 [000] d... 7269834.882517: block_rq_insert: 202,1 R 0 () 736304 + 256 [cksum]\n           cksum-11908 [000] d... 7269834.882528: <stack trace>\n => __elv_add_request\n => blk_flush_plug_list\n => blk_finish_plug\n => __do_page_cache_readahead\n => ondemand_readahead\n => page_cache_sync_readahead\n => generic_file_read_iter\n => new_sync_read\n => vfs_read\n => SyS_read\n => system_call_fastpath\n[...]\n\n\nYou can also add tracepoint filters. To see what variables you can use, use -v:\n\n# ./tpoint -v block:block_rq_issue \nname: block_rq_issue\nID: 942\nformat:\n\tfield:unsigned short common_type;\toffset:0;\tsize:2;\tsigned:0;\n\tfield:unsigned char common_flags;\toffset:2;\tsize:1;\tsigned:0;\n\tfield:unsigned char common_preempt_count;\toffset:3;\tsize:1;\tsigned:0;\n\tfield:int common_pid;\toffset:4;\tsize:4;\tsigned:1;\n\n\tfield:dev_t dev;\toffset:8;\tsize:4;\tsigned:0;\n\tfield:sector_t sector;\toffset:16;\tsize:8;\tsigned:0;\n\tfield:unsigned int nr_sector;\toffset:24;\tsize:4;\tsigned:0;\n\tfield:unsigned int bytes;\toffset:28;\tsize:4;\tsigned:0;\n\tfield:char rwbs[8];\toffset:32;\tsize:8;\tsigned:1;\n\tfield:char comm[16];\toffset:40;\tsize:16;\tsigned:1;\n\tfield:__data_loc char[] cmd;\toffset:56;\tsize:4;\tsigned:1;\n\nprint fmt: \"%d,%d %s %u (%s) %llu + %u [%s]\", ((unsigned int) ((REC->dev) >> 20)), ((unsigned int) ((REC->dev) & ((1U << 20) - 1))), REC->rwbs, REC->bytes, __get_str(cmd), (unsigned long long)REC->sector, REC->nr_sector, REC->comm\n\n\nNow I'll add a filter to check that the rwbs field (I/O type) includes an \"R\",\nmaking it a read:\n\n# ./tpoint -s block:block_rq_insert 'rwbs ~ \"*R*\"'\n           cksum-11908 [000] d... 7269839.919098: block_rq_insert: 202,1 R 0 () 736560 + 136 [cksum]\n           cksum-11908 [000] d... 7269839.919107: <stack trace>\n => __elv_add_request\n => blk_flush_plug_list\n => blk_finish_plug\n => __do_page_cache_readahead\n => ondemand_readahead\n => page_cache_async_readahead\n => generic_file_read_iter\n => new_sync_read\n => vfs_read\n => SyS_read\n => system_call_fastpath\n[...]\n\n\nUse -h to print the USAGE message:\n\n# ./tpoint -h\nUSAGE: tpoint [-hHsv] [-d secs] [-p PID] [-L TID] tracepoint [filter]\n       tpoint -l\n                 -d seconds      # trace duration, and use buffers\n                 -p PID          # PID to match on events\n                 -L TID          # thread id to match on events\n                 -v              # view format file (don't trace)\n                 -H              # include column headers\n                 -l              # list all tracepoints\n                 -s              # show kernel stack traces\n                 -h              # this usage message\n\nNote that these examples may need modification to match your kernel\nversion's function names and platform's register usage.\n   eg,\n       tpoint -l | grep open\n                                 # find tracepoints containing \"open\"\n       tpoint syscalls:sys_enter_open\n                                 # trace open() syscall entry\n       tpoint block:block_rq_issue\n                                 # trace block I/O issue\n       tpoint -s block:black_rq_issue\n                                 # show kernel stacks\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "examples/uprobe_example.txt",
    "content": "Demonstrations of uprobe, the Linux ftrace version.\n\nTrace the readline() function from all processes named \"bash\":\n\n# ./uprobe p:bash:readline\nTracing uprobe readline (p:readline /bin/bash:0x8db60). Ctrl-C to end.\n            bash-11886 [003] d... 19601233.618462: readline: (0x48db60)\n            bash-11886 [003] d... 19601235.152067: readline: (0x48db60)\n            bash-11915 [003] d... 19601238.976244: readline: (0x48db60)\n^C\nEnding tracing...\n\nreadline() is the bash shell's function for reading interactive input, and\na line is printed each time I entered commands in separate bash shells.\nThe line contains default ftrace columns: the process name, \"-\", and PID;\nthe CPU, flags, a timestamp (in units of seconds), the probe name, then\nother arguments. These columns are documented in the kernel source, under\nDocumentation/trace/ftrace.txt.\n\nThe first line of output is informational, and shows what uprobe is really\ndoing: it turned \"bash\" into \"/bin/bash\", using a $PATH lookup (via which(1)).\nIt then turned the \"readline\" symbol into 0x8db60, using objdump(1) for\nsymbol lookups.\n\nNote that this traces _all_ bash processes simultaneously.\n\n\nTracing PID 11886 only:\n\n# ./uprobe -p 11886 p:bash:readline\nTracing uprobe readline (p:readline /bin/bash:0x8db60). Ctrl-C to end.\n            bash-11886 [002] d... 19601657.753893: readline: (0x48db60)\n            bash-11886 [002] d... 19601658.246613: readline: (0x48db60)\n            bash-11886 [002] d... 19601658.386666: readline: (0x48db60)\n            bash-11886 [002] d... 19601661.415952: readline: (0x48db60)\n^C\nEnding tracing...\n\nThis may be important if you are tracing shared library functions, and only care\nabout one target process.\n\n\nYou can specify the full path to a binary to trace:\n\n# ./uprobe p:/bin/bash:readline\nTracing uprobe readline (p:readline /bin/bash:0x8db60). Ctrl-C to end.\n            bash-11886 [002] d... 19601746.902461: readline: (0x48db60)\n            bash-11886 [002] d... 19601749.543485: readline: (0x48db60)\n            bash-11886 [001] d... 19601749.702369: readline: (0x48db60)\n^C\nEnding tracing...\n\nThis might be useful if uprobe picked the wrong binary to trace, as shown by\nthe informational line, and you wanted to specify it directly. It is also useful\nfor tracing binaries not in the $PATH, which uprobe can't otherwise find.\n\n\nUse -l to list symbols available to trace; eg, searching for functions\ncontaining \"readline\" in bash:\n\n# ./uprobe -l bash | grep readline\ninitialize_readline\npcomp_set_readline_variables\nposix_readline_initialize\nreadline\nreadline_internal_char\nreadline_internal_setup\nreadline_internal_teardown\n\n\nTracing the return of readline() with return value as a string:\n\n# ./uprobe 'r:bash:readline +0($retval):string'\nTracing uprobe readline (r:readline /bin/bash:0x8db60 +0($retval):string). Ctrl-C to end.\n            bash-11886 [003] d... 19601837.001935: readline: (0x41e876 <- 0x48db60) arg1=\"ls -l\"\n            bash-11886 [002] d... 19601851.008409: readline: (0x41e876 <- 0x48db60) arg1=\"echo \"hello world\"\"\n            bash-11886 [002] d... 19601854.099730: readline: (0x41e876 <- 0x48db60) arg1=\"df -h\"\n            bash-11886 [002] d... 19601858.805740: readline: (0x41e876 <- 0x48db60) arg1=\"cd ..\"\n            bash-11886 [003] d... 19601898.378753: readline: (0x41e876 <- 0x48db60) arg1=\"foo bar\"\n^C\nEnding tracing...\n\nNow I can see the commands entered. Note that this traces what bash reads in,\neven if the command eventually fails. Eg, the last command \"foo bar\" didn't\nwork (No command 'foo' found).\n\nNote that this invocation now uses \"r:\" at the start of the probe description,\ninstead of \"p:\". r is for return probes, p for entry probes.\n\n\nTracing sleep() calls in all running libc shared libraries:\n\n# ./uprobe p:libc:sleep\nTracing uprobe sleep (p:sleep /lib/x86_64-linux-gnu/libc-2.15.so:0xbf130). Ctrl-C to end.\n          svscan-2134  [000] d... 19602402.959904: sleep: (0x7f2dba562130)\n            cron-923   [000] d... 19602404.640507: sleep: (0x7f3e26d9e130)\n            cron-923   [002] d... 19602404.655232: sleep: (0x7f3e26d9e130)\n            cron-923   [002] d... 19602405.189271: sleep: (0x7f3e26d9e130)\n          svscan-2134  [000] d... 19602407.959947: sleep: (0x7f2dba562130)\n[...]\n\nThis shows different programs calling sleep -- likely threads waiting for work.\n\nI ran a \"sleep 1\" command in a bash shell, which wasn't seen above: probably\nusing a different sleep library call, which I'd need to trace separately.\n\n\nIncluding headers (-H):\n\n# ./uprobe -H p:libc:sleep\nTracing uprobe sleep (p:sleep /lib/x86_64-linux-gnu/libc-2.15.so:0xbf130). Ctrl-C to end.\n# tracer: nop\n#\n# entries-in-buffer/entries-written: 0/0   #P:4\n#\n#                              _-----=> irqs-off\n#                             / _----=> need-resched\n#                            | / _---=> hardirq/softirq\n#                            || / _--=> preempt-depth\n#                            ||| /     delay\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#              | |       |   ||||       |         |\n          svscan-2134  [000] d... 19603052.976770: sleep: (0x7f2dba562130)\n          svscan-2134  [002] d... 19603057.976927: sleep: (0x7f2dba562130)\n[...]\n\nThese are documented in Documentation/trace/ftrace.txt.\n\n\nTracing sleep() with its argument (seconds):\n\n# ./uprobe 'p:libc:sleep %di'\nTracing uprobe sleep (p:sleep /lib/x86_64-linux-gnu/libc-2.15.so:0xbf130 %di). Ctrl-C to end.\n          svscan-2134  [002] d... 19602517.962925: sleep: (0x7f2dba562130) arg1=0x5\n          svscan-2134  [002] d... 19602522.963082: sleep: (0x7f2dba562130) arg1=0x5\n            cron-923   [002] d... 19602524.187733: sleep: (0x7f3e26d9e130) arg1=0x3c\n          svscan-2134  [002] d... 19602527.963267: sleep: (0x7f2dba562130) arg1=0x5\n[...]\n\nSo svcan was sleeping for 5 seconds, and cron for 60 seconds (0x3c = 60).\n\nThe argument is specified by its register, %di. This is platform dependent: %di\nmay only be meaningful on x86. If you're on a different architecture (eg, ARM),\nyou will probably need to use something else.\n\nIf working with registers is not for you, then consider tracing this using\nperf_events with debuginfo installed: in which case you can use the variable\nnames. Or consider a different tracer.\n\n\nHere is an example of the optional filter expression, to only trace the return\nof fopen() when it failed and returned NULL (0):\n\n# ./uprobe 'r:libc:fopen file=$retval' 'file == 0'\nTracing uprobe fopen (r:fopen /lib/x86_64-linux-gnu/libc-2.15.so:0x6e540 file=$retval). Ctrl-C to end.\n           prog1-23982 [000] d... 19602894.346872: fopen: (0x40051e <- 0x7f637867f540) file=0x0\n^C\nEnding tracing...\n\nThe argument $retval was given a vanity name \"file\", which was then tested in\nthe filter expression \"file == 0\".\n\n\nHere's an example of tracing the MySQL server dispatch_command() function, along\nwith the query string (note: the %dx register is only valid for this\narchitecture and this software build):\n\n# ./uprobe 'p:dispatch_command /opt/mysql/bin/mysqld:_Z16dispatch_command19enum_server_commandP3THDPcj +0(%dx):string'\nTracing uprobe dispatch_command (p:dispatch_command /opt/mysql/bin/mysqld:0x2dbd40 +0(%dx):string). Ctrl-C to end.\n          mysqld-2855  [001] d... 19956674.509085: dispatch_command: (0x6dbd40) arg1=\"show tables\"\n          mysqld-2855  [001] d... 19956675.541155: dispatch_command: (0x6dbd40) arg1=\"SELECT * FROM numbers where number > 32000\"\n^C\nEnding tracing...\n\nThe function name, \"_Z16dispatch_command19enum_server_commandP3THDPcj\", is the\nC++ mangled symbol.\n\nI can name the query string argument \"cmd\" then test it in a filter; eg, to only\nmatch queries that begin with \"SELECT\":\n\n# ./uprobe 'p:dispatch_command /opt/mysql/bin/mysqld:_Z16dispatch_command19enum_server_commandP3THDPcj cmd=+0(%dx):string' 'cmd ~ \"SELECT*\"'\nTracing uprobe dispatch_command (p:dispatch_command /opt/mysql/bin/mysqld:0x2dbd40 cmd=+0(%dx):string). Ctrl-C to end.\n          mysqld-2855  [001] d... 19956754.619958: dispatch_command: (0x6dbd40) cmd=\"SELECT * FROM numbers where number > 32000\"\n          mysqld-2855  [001] d... 19956755.060125: dispatch_command: (0x6dbd40) cmd=\"SELECT * FROM numbers where number > 32000\"\n^C\nEnding tracing...\n\n\nOverhead is relative to the rate of events: a higher rate of traced events,\nmeans uprobe costs higher overhead. If you are unsure of the rate of events,\nyou can capture a set number only, or trace for a limited duration only (covered\nin the next example). To trace a set number only, you can pipe into head, eg:\n\n# ./uprobe -p 11982 p:bash:sh_malloc | head -15\nTracing uprobe sh_malloc (p:sh_malloc /bin/bash:0xaafa0). Ctrl-C to end.\n            bash-11982 [001] d... 19643121.529484: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529493: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529506: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529510: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529519: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529521: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529523: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529525: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529531: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529533: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529536: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529541: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529546: sh_malloc: (0x4aafa0)\n            bash-11982 [001] d... 19643121.529549: sh_malloc: (0x4aafa0)\n\nuprobe traps SIGPIPE, so that it properly exits and cleans up probes when used\nin this fashion.\n\nNote the timestamps: by examining the rate they are increasing, you can have\nsome estimation for the rate of events. In this case, the 15 events all\nhappened within the same millisecond (the timestamp column is in units of\nseconds), which suggests these are frequent events.\n\n\nThe -d option can be used to specify a duration for tracing, which also causes\nuprobe to perform in-kernel buffering, which reduces the overhead of tracing:\n\n# ./uprobe -d 5 p:libc:gettimeofday\nTracing uprobe gettimeofday for 5 seconds (buffered)...\n           sleep-12743 [001] d... 19642858.943440: gettimeofday: (0x7f400138ac10)\n       rotatelog-12744 [000] d... 19642858.955665: gettimeofday: (0x7f0ba34ebc10)\n       rotatelog-12745 [003] d... 19642858.956425: gettimeofday: (0x7f1e6db20c10)\n       rotatelog-12744 [000] d... 19642858.956924: gettimeofday: (0x7f0ba34ebc10)\n       rotatelog-12745 [003] d... 19642858.957608: gettimeofday: (0x7f1e6db20c10)\n       rotatelog-12744 [001] d... 19642858.958005: gettimeofday: (0x7fd8a1d64c10)\n       rotatelog-12744 [003] d... 19642858.959496: gettimeofday: (0x7f9531acdc10)\n           mkdir-12746 [002] d... 19642858.959542: gettimeofday: (0x7fd539474c10)\n           chown-12747 [001] d... 19642858.961455: gettimeofday: (0x7ff5646afc10)\n       rotatelog-12745 [000] d... 19642858.963065: gettimeofday: (0x7f406aca7c10)\n       rotatelog-12745 [001] d... 19642858.964280: gettimeofday: (0x7f6548debc10)\n       rotatelog-12749 [000] d... 19642859.977462: gettimeofday: (0x7fecaf7e1c10)\n       rotatelog-12750 [003] d... 19642859.977697: gettimeofday: (0x7f821eb3cc10)\n       rotatelog-12749 [000] d... 19642859.978707: gettimeofday: (0x7fecaf7e1c10)\n[...]\n\nYou will not see live output during the -d mode, as it is being buffered\nin-kernel.\n\n\nTracing func_abc() in my test program, and including user-level stacks:\n\n# ./uprobe -s p:/root/func_abc:func_c\nTracing uprobe func_c (p:func_c /root/func_abc:0x4f4). Ctrl-C to end.\n        func_abc-25394 [000] d... 19603250.054040: func_c: (0x4004f4)\n        func_abc-25394 [000] d... 19603250.054056: <user stack trace>\n =>  <00000000004004f4>\n =>  <0000000000400527>\n =>  <0000000000400537>\n =>  <00007fca9f0e376d>\n        func_abc-25394 [000] d... 19603251.054250: func_c: (0x4004f4)\n        func_abc-25394 [000] d... 19603251.054266: <user stack trace>\n =>  <00000000004004f4>\n =>  <0000000000400527>\n =>  <0000000000400537>\n =>  <00007fca9f0e376d>\n^C\nEnding tracing...\n\nThe output has the raw hex addresses. If this is too much of a nuisance, then\ntry tracing this using perf_events which should automate the translation.\n\nIt can get worse, eg:\n\nl# ./uprobe -s p:bash:readline\nTracing uprobe readline (p:readline /bin/bash:0x8db60). Ctrl-C to end.\n            bash-11886 [002] d... 19603434.397818: readline: (0x48db60)\n            bash-11886 [002] d... 19603434.397832: <user stack trace>\n =>  <000000000048db60>\n            bash-11886 [002] d... 19603434.592500: readline: (0x48db60)\n            bash-11886 [002] d... 19603434.592510: <user stack trace>\n =>  <000000000048db60>\n^C\nEnding tracing...\n\nHere the stack trace is missing (0x48db60 is the traced function, transposed\nfrom the base load address). This is due to compiler optimizations. It can be\nfixed by recompiling with -fno-omit-frame-pointer, or, using perf_events and\na different method of stack walking.\n\n\nUse -h to print the USAGE message:\n\n# ./uprobe -h\nUSAGE: uprobe [-FhHsv] [-d secs] [-p PID] [-L TID] {-l target |\n              uprobe_definition [filter]}\n                 -F              # force. trace despite warnings.\n                 -d seconds      # trace duration, and use buffers\n                 -l target       # list functions from this executable\n                 -p PID          # PID to match on events\n                 -L TID          # thread id to match on events\n                 -v              # view format file (don't trace)\n                 -H              # include column headers\n                 -s              # show user stack traces\n                 -h              # this usage message\n\nNote that these examples may need modification to match your kernel\nversion's function names and platform's register usage.\n   eg,\n       # trace readline() calls in all running \"bash\" executables:\n           uprobe p:bash:readline\n       # trace readline() with explicit executable path:\n           uprobe p:/bin/bash:readline\n       # trace the return of readline() with return value as a string:\n           uprobe 'r:bash:readline +0($retval):string'\n       # trace sleep() calls in all running libc shared libraries:\n           uprobe p:libc:sleep\n       # trace sleep() with register %di (x86):\n           uprobe 'p:libc:sleep %di'\n       # trace this address (use caution: must be instruction aligned):\n           uprobe p:libc:0xbf130\n       # trace gettimeofday() for PID 1182 only:\n           uprobe -p 1182 p:libc:gettimeofday\n       # trace the return of fopen() only when it returns NULL:\n           uprobe 'r:libc:fopen file=$retval' 'file == 0'\n\nSee the man page and example file for more info.\n"
  },
  {
    "path": "execsnoop",
    "content": "#!/bin/bash\n#\n# execsnoop - trace process exec() with arguments.\n#             Written using Linux ftrace.\n#\n# This shows the execution of new processes, especially short-lived ones that\n# can be missed by sampling tools such as top(1).\n#\n# USAGE: ./execsnoop [-hrt] [-n name]\n#\n# REQUIREMENTS: FTRACE and KPROBE CONFIG, sched:sched_process_fork tracepoint,\n# and either the sys_execve, stub_execve or do_execve kernel function. You may\n# already have these on recent kernels. And awk.\n#\n# This traces exec() from the fork()->exec() sequence, which means it won't\n# catch new processes that only fork(). With the -r option, it will also catch\n# processes that re-exec. It makes a best-effort attempt to retrieve the program\n# arguments and PPID; if these are unavailable, 0 and \"[?]\" are printed\n# respectively. There is also a limit to the number of arguments printed (by\n# default, 8), which can be increased using -a.\n#\n# This implementation is designed to work on older kernel versions, and without\n# kernel debuginfo. It works by dynamic tracing an execve kernel function to\n# read the arguments from the %si register. The sys_execve function is tried\n# first, then stub_execve and do_execve. The sched:sched_process_fork\n# tracepoint is used to get the PPID. This program is a workaround that should be\n# improved in the future when other kernel capabilities are made available. If\n# you need a more reliable tool now, then consider other tracing alternatives\n# (eg, SystemTap). This tool is really a proof of concept to see what ftrace can\n# currently do.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the execsnoop(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 07-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock; wroteflock=0\nopt_duration=0; duration=; opt_name=0; name=; opt_time=0; opt_reexec=0\nopt_argc=0; argc=8; max_argc=16; ftext=\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: execsnoop [-hrt] [-a argc] [-d secs] [name]\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -a argc         # max args to show (default 8)\n\t                 -r              # include re-execs\n\t                 -t              # include time (seconds)\n\t                 -h              # this usage message\n\t                 name            # process name to match (REs allowed)\n\t  eg,\n\t       execsnoop                 # watch exec()s live (unbuffered)\n\t       execsnoop -d 1            # trace 1 sec (buffered)\n\t       execsnoop grep            # trace process names containing grep\n\t       execsnoop 'udevd$'        # process names ending in \"udevd\"\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > events/kprobes/$kname/enable\"\n\twarn \"echo 0 > events/sched/sched_process_fork/enable\"\n\twarn \"echo -:$kname >> kprobe_events\"\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts a:d:hrt opt\ndo\n\tcase $opt in\n\ta)\topt_argc=1; argc=$OPTARG ;;\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tr)\topt_reexec=1 ;;\n\tt)\topt_time=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\nif (( $# )); then\n\topt_name=1\n\tname=$1\n\tshift\nfi\n(( $# )) && usage\n\n### option logic\n(( opt_pid && opt_name )) && die \"ERROR: use either -p or -n.\"\n(( opt_pid )) && ftext=\" issued by PID $pid\"\n(( opt_name )) && ftext=\" issued by process name \\\"$name\\\"\"\n(( opt_file )) && ftext=\"$ftext for filenames containing \\\"$file\\\"\"\n(( opt_argc && argc > max_argc )) && die \"ERROR: max -a argc is $max_argc.\"\nif (( opt_duration )); then\n\techo \"Tracing exec()s$ftext for $duration seconds (buffered)...\"\nelse\n\techo \"Tracing exec()s$ftext. Ctrl-C to end.\"\nfi\n\n### select awk\nif (( opt_duration )); then\n\t[[ -x /usr/bin/mawk ]] && awk=mawk || awk=awk\nelse\n\t# workarounds for mawk/gawk fflush behavior\n\tif [[ -x /usr/bin/gawk ]]; then\n\t\tawk=gawk\n\telif [[ -x /usr/bin/mawk ]]; then\n\t\tawk=\"mawk -W interactive\"\n\telse\n\t\tawk=awk\n\tfi\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### build probe\nif [[ -x /usr/bin/getconf ]]; then\n\tbits=$(getconf LONG_BIT)\nelse\n\tbits=64\n\t[[ $(uname -m) == i* ]] && bits=32\nfi\n(( offset = bits / 8 ))\nfunction makeprobe {\n\tfunc=$1\n\tkname=execsnoop_$func\n\tkprobe=\"p:$kname $func\"\n\ti=0\n\twhile (( i < argc + 1 )); do\n\t\t# p:kname do_execve +0(+0(%si)):string +0(+8(%si)):string ...\n\t\tkprobe=\"$kprobe +0(+$(( i * offset ))(%si)):string\"\n\t\t(( i++ ))\n\tdone\n}\n# try in this order: sys_execve, stub_execve, do_execve\nmakeprobe sys_execve\n\n### setup and begin tracing\necho nop > current_tracer\nif ! echo $kprobe >> kprobe_events 2>/dev/null; then\n\tmakeprobe stub_execve\n\tif ! echo $kprobe >> kprobe_events 2>/dev/null; then\n\t    makeprobe do_execve\n\t    if ! echo $kprobe >> kprobe_events 2>/dev/null; then\n\t\t    edie \"ERROR: adding a kprobe for execve. Exiting.\"\n        fi\n\tfi\nfi\nif ! echo 1 > events/kprobes/$kname/enable; then\n\tedie \"ERROR: enabling kprobe for execve. Exiting.\"\nfi\nif ! echo 1 > events/sched/sched_process_fork/enable; then\n\tedie \"ERROR: enabling sched:sched_process_fork tracepoint. Exiting.\"\nfi\necho \"Instrumenting $func\"\n(( opt_time )) && printf \"%-16s \" \"TIMEs\"\nprintf \"%6s %6s %s\\n\" \"PID\" \"PPID\" \"ARGS\"\n\n#\n# Determine output format. It may be one of the following (newest first):\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#           TASK-PID    CPU#    TIMESTAMP  FUNCTION\n# To differentiate between them, the number of header fields is counted,\n# and an offset set, to skip the extra column when needed.\n#\noffset=$($awk 'BEGIN { o = 0; }\n\t$1 == \"#\" && $2 ~ /TASK/ && NF == 6 { o = 1; }\n\t$2 ~ /TASK/ { print o; exit }' trace)\n\n### print trace buffer\nwarn \"echo > trace\"\n( if (( opt_duration )); then\n\t# wait then dump buffer\n\tsleep $duration\n\tcat -v trace\nelse\n\t# print buffer live\n\tcat -v trace_pipe\nfi ) | $awk -v o=$offset -v opt_name=$opt_name -v name=$name \\\n    -v opt_duration=$opt_duration -v opt_time=$opt_time -v kname=$kname \\\n    -v opt_reexec=$opt_reexec '\n\t# common fields\n\t$1 != \"#\" {\n\t\t# task name can contain dashes\n\t\tcomm = pid = $1\n\t\tsub(/-[0-9][0-9]*/, \"\", comm)\n\t\tsub(/.*-/, \"\", pid)\n\t}\n\n\t$1 != \"#\" && $(4+o) ~ /sched_process_fork/ {\n\t\tcpid=$0\n\t\tsub(/.* child_pid=/, \"\", cpid)\n\t\tsub(/ .*/, \"\", cpid)\n\t\tgetppid[cpid] = pid\n\t\tdelete seen[pid]\n\t}\n\n\t$1 != \"#\" && $(4+o) ~ kname {\n\t\tif (seen[pid])\n\t\t\tnext\n\t\tif (opt_name && comm !~ name)\n\t\t\tnext\n\n\t\t#\n\t\t# examples:\n\t\t# ... arg1=\"/bin/echo\" arg2=\"1\" arg3=\"2\" arg4=\"3\" ...\n\t\t# ... arg1=\"sleep\" arg2=\"2\" arg3=(fault) arg4=\"\" ...\n\t\t# ... arg1=\"\" arg2=(fault) arg3=\"\" arg4=\"\" ...\n\t\t# the last example is uncommon, and may be a race.\n\t\t#\n\t\tif ($0 ~ /arg1=\"\"/) {\n\t\t\targs = comm \" [?]\"\n\t\t} else {\n\t\t\targs=$0\n\t\t\tsub(/ arg[0-9]*=\\(fault\\).*/, \"\", args)\n\t\t\tsub(/.*arg1=\"/, \"\", args)\n\t\t\tgsub(/\" arg[0-9]*=\"/, \" \", args)\n\t\t\tsub(/\"$/, \"\", args)\n\t\t\tif ($0 !~ /\\(fault\\)/)\n\t\t\t\targs = args \" [...]\"\n\t\t}\n\n\t\tif (opt_time) {\n\t\t\ttime = $(3+o); sub(\":\", \"\", time)\n\t\t\tprintf \"%-16s \", time\n\t\t}\n\t\tprintf \"%6s %6d %s\\n\", pid, getppid[pid], args\n\t\tif (!opt_duration)\n\t\t\tfflush()\n\t\tif (!opt_reexec) {\n\t\t\tseen[pid] = 1\n\t\t\tdelete getppid[pid]\n\t\t}\n\t}\n\n\t$0 ~ /LOST.*EVENT[S]/ { print \"WARNING: \" $0 > \"/dev/stderr\" }\n'\n\n### end tracing\nend\n"
  },
  {
    "path": "fs/cachestat",
    "content": "#!/bin/bash\n#\n# cachestat - show Linux page cache hit/miss statistics.\n#             Uses Linux ftrace.\n#\n# This is a proof of concept using Linux ftrace capabilities on older kernels,\n# and works by using function profiling for in-kernel counters. Specifically,\n# four kernel functions are traced:\n#\n#\tmark_page_accessed() for measuring cache accesses\n#\tmark_buffer_dirty() for measuring cache writes\n#\tadd_to_page_cache_lru() for measuring page additions\n#\taccount_page_dirtied() for measuring page dirties\n#\n# It is possible that these functions have been renamed (or are different\n# logically) for your kernel version, and this script will not work as-is.\n# This script was written on Linux 3.13. This script is a sandcastle: the\n# kernel may wash some away, and you'll need to rebuild.\n#\n# USAGE: cachestat [-Dht] [interval]\n#    eg,\n#        cachestat 5\t# show stats every 5 seconds\n#\n# Run \"cachestat -h\" for full usage.\n#\n# WARNING: This uses dynamic tracing of kernel functions, and could cause\n# kernel panics or freezes. Test, and know what you are doing, before use.\n# It also traces cache activity, which can be frequent, and cost some overhead.\n# The statistics should be treated as best-effort: there may be some error\n# margin depending on unusual workload types.\n#\n# REQUIREMENTS: CONFIG_FUNCTION_PROFILER, awk.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 28-Dec-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\ninterval=1; opt_timestamp=0; opt_debug=0\ntrap 'quit=1' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: cachestat [-Dht] [interval]\n\t                 -D              # print debug counters\n\t                 -h              # this usage message\n\t                 -t              # include timestamp\n\t                 interval        # output interval in secs (default 1)\n\t  eg,\n\t       cachestat                 # show stats every second\n\t       cachestat 5               # show stats every 5 seconds\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\n### process options\nwhile getopts Dht opt\ndo\n\tcase $opt in\n\tD)\topt_debug=1 ;;\n\tt)\topt_timestamp=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### option logic\nif (( $# )); then\n\tinterval=$1\nfi\necho \"Counting cache functions... Output every $interval seconds.\"\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### enable tracing\nsysctl -q kernel.ftrace_enabled=1\t# doesn't set exit status\nprintf \"mark_page_accessed\\nmark_buffer_dirty\\nadd_to_page_cache_lru\\naccount_page_dirtied\\n\" > set_ftrace_filter || \\\n    die \"ERROR: tracing these four kernel functions: mark_page_accessed,\"\\\n    \"mark_buffer_dirty, add_to_page_cache_lru and account_page_dirtied (unknown kernel version?). Exiting.\"\nwarn \"echo nop > current_tracer\"\nif ! echo 1 > function_profile_enabled; then\n\techo > set_ftrace_filter\n\tdie \"ERROR: enabling function profiling. Have CONFIG_FUNCTION_PROFILER? Exiting.\"\nfi\n\n(( opt_timestamp )) && printf \"%-8s \" TIME\nprintf \"%8s %8s %8s %8s %12s %10s\" HITS MISSES DIRTIES RATIO \"BUFFERS_MB\" \"CACHE_MB\"\n(( opt_debug )) && printf \"  DEBUG\"\necho\n\n### summarize\nquit=0; secs=0\nwhile (( !quit && (!opt_duration || secs < duration) )); do\n\t(( secs += interval ))\n\techo 0 > function_profile_enabled\n\techo 1 > function_profile_enabled\n\tsleep $interval\n\n\t(( opt_timestamp )) && printf \"%(%H:%M:%S)T \" -1\n\n\t# cat both meminfo and trace stats, and let awk pick them apart\n\tcat /proc/meminfo trace_stat/function* | awk -v debug=$opt_debug '\n\t# match meminfo stats:\n\t$1 == \"Buffers:\" && $3 == \"kB\" { buffers_mb = $2 / 1024 }\n\t$1 == \"Cached:\" && $3 == \"kB\" { cached_mb = $2 / 1024 }\n\t# identify and save trace counts:\n\t$2 ~ /[0-9]/ && $3 != \"kB\" { a[$1] += $2 }\n\tEND {\n\t\tmpa = a[\"mark_page_accessed\"]\n\t\tmbd = a[\"mark_buffer_dirty\"]\n\t\tapcl = a[\"add_to_page_cache_lru\"]\n\t\tapd = a[\"account_page_dirtied\"]\n\n\t\ttotal = mpa - mbd\n\t\tmisses = apcl - apd\n\t\tif (misses < 0)\n\t\t\tmisses = 0\n\t\thits = total - misses\n\n\t\tratio = 100 * hits / total\n\t\tprintf \"%8d %8d %8d %7.1f%% %12.0f %10.0f\", hits, misses, mbd,\n\t\t    ratio, buffers_mb, cached_mb\n\t\tif (debug)\n\t\t\tprintf \"  (%d %d %d %d)\", mpa, mbd, apcl, apd\n\t\tprintf \"\\n\"\n\t}'\ndone\n\n### end tracing\necho 2>/dev/null\necho \"Ending tracing...\" 2>/dev/null\nwarn \"echo 0 > function_profile_enabled\"\nwarn \"echo > set_ftrace_filter\"\n"
  },
  {
    "path": "iolatency",
    "content": "#!/bin/bash\n#\n# iolatency - summarize block device I/O latency as a histogram.\n#             Written using Linux ftrace.\n#\n# This shows the distribution of latency, allowing modes and latency outliers\n# to be identified and studied.\n#\n# USAGE: ./iolatency [-hQT] [-d device] [-i iotype] [interval [count]]\n#\n# REQUIREMENTS: FTRACE CONFIG and block:block_rq_* tracepoints, which you may\n# already have on recent kernels.\n#\n# OVERHEAD: block device I/O issue and completion events are traced and buffered\n# in-kernel, then processed and summarized in user space. There may be\n# measurable overhead with this approach, relative to the block device IOPS.\n#\n# This was written as a proof of concept for ftrace.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 20-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock\nbufsize_kb=4096\nopt_device=0; device=; opt_iotype=0; iotype=; opt_timestamp=0\nopt_interval=0; interval=1; opt_count=0; count=0; opt_queue=0\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: iolatency [-hQT] [-d device] [-i iotype] [interval [count]]\n\t                 -d device       # device string (eg, \"202,1)\n\t                 -i iotype       # match type (eg, '*R*' for all reads)\n\t                 -Q              # use queue insert as start time\n\t                 -T              # timestamp on output\n\t                 -h              # this usage message\n\t                 interval        # summary interval, seconds (default 1)\n\t                 count           # number of summaries\n\t  eg,\n\t       iolatency                 # summarize latency every second\n\t       iolatency -Q              # include block I/O queue time\n\t       iolatency 5 2             # 2 x 5 second summaries\n\t       iolatency -i '*R*'        # trace reads\n\t       iolatency -d 202,1        # trace device 202,1 only\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > events/block/$b_start/enable\"\n\twarn \"echo 0 > events/block/block_rq_complete/enable\"\n\tif (( opt_device || opt_iotype )); then\n\t\twarn \"echo 0 > events/block/$b_start/filter\"\n\t\twarn \"echo 0 > events/block/block_rq_complete/filter\"\n\tfi\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts d:hi:QT opt\ndo\n\tcase $opt in\n\td)\topt_device=1; device=$OPTARG ;;\n\ti)\topt_iotype=1; iotype=$OPTARG ;;\n\tQ)\topt_queue=1 ;;\n\tT)\topt_timestamp=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\nif (( $# )); then\n\topt_interval=1\n\tinterval=$1\n\tshift\nfi\nif (( $# )); then\n\topt_count=1\n\tcount=$1\nfi\nif (( opt_device )); then\n\tmajor=${device%,*}\n\tminor=${device#*,}\n\tdev=$(( (major << 20) + minor ))\nfi\nif (( opt_queue )); then\n\tb_start=block_rq_insert\nelse\n\tb_start=block_rq_issue\nfi\n\n### select awk\n[[ -x /usr/bin/mawk ]] && awk='mawk -W interactive' || awk=awk\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and begin tracing\nwarn \"echo nop > current_tracer\"\nwarn \"echo $bufsize_kb > buffer_size_kb\"\nfilter=\nif (( opt_iotype )); then\n\tfilter=\"rwbs ~ \\\"$iotype\\\"\"\nfi\nif (( opt_device )); then\n\t[[ \"$filter\" != \"\" ]] && filter=\"$filter && \"\n\tfilter=\"${filter}dev == $dev\"\nfi\nif (( opt_iotype || opt_device )); then\n\tif ! echo \"$filter\" > events/block/$b_start/filter || \\\n\t    ! echo \"$filter\" > events/block/block_rq_complete/filter\n\tthen\n\t\tedie \"ERROR: setting -d or -t filter. Exiting.\"\n\tfi\nfi\nif ! echo 1 > events/block/$b_start/enable || \\\n    ! echo 1 > events/block/block_rq_complete/enable; then\n\tedie \"ERROR: enabling block I/O tracepoints. Exiting.\"\nfi\netext=\n(( !opt_count )) && etext=\" Ctrl-C to end.\"\necho \"Tracing block I/O. Output every $interval seconds.$etext\"\n\n#\n# Determine output format. It may be one of the following (newest first):\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#           TASK-PID    CPU#    TIMESTAMP  FUNCTION\n# To differentiate between them, the number of header fields is counted,\n# and an offset set, to skip the extra column when needed.\n#\noffset=$($awk 'BEGIN { o = 0; }\n\t$1 == \"#\" && $2 ~ /TASK/ && NF == 6 { o = 1; }\n\t$2 ~ /TASK/ { print o; exit }' trace)\n\n### print trace buffer\nwarn \"echo > trace\"\ni=0\nwhile (( !opt_count || (i < count) )); do\n\t(( i++ ))\n\tsleep $interval\n\n\t# snapshots were added in 3.10\n\tif [[ -x snapshot ]]; then\n\t\techo 1 > snapshot\n\t\techo > trace\n\t\tcat snapshot\n\telse\n\t\tcat trace\n\t\techo > trace\n\tfi\n\n\t(( opt_timestamp )) && printf \"time %(%H:%M:%S)T:\\n\" -1\n\techo \"tick\"\ndone | \\\n$awk -v o=$offset -v opt_timestamp=$opt_timestamp -v b_start=$b_start '\n\tfunction star(sval, smax, swidth) {\n\t\tstars = \"\"\n\t\tif (smax == 0) return \"\"\n\t\tfor (si = 0; si < (swidth * sval / smax); si++) {\n\t\t\tstars = stars \"#\"\n\t\t}\n\t\treturn stars\n\t}\n\n\tBEGIN { max_i = 0 }\n\n\t# common fields\n\t$1 != \"#\" {\n\t\ttime = $(3+o); sub(\":\", \"\", time)\n\t\tdev = $(5+o)\n\t}\n\n\t# block I/O request\n\t$1 != \"#\" && $0 ~ b_start {\n\t\t#\n\t\t# example: (fields1..4+o) 202,1 W 0 () 12862264 + 8 [tar]\n\t\t# The cmd field \"()\" might contain multiple words (hex),\n\t\t# hence stepping from the right (NF-3).\n\t\t#\n\t\tloc = $(NF-3)\n\t\tstarts[dev, loc] = time\n\t\tnext\n\t}\n\n\t# block I/O completion\n\t$1 != \"#\" && $0 ~ /rq_complete/ {\n\t\t#\n\t\t# example: (fields1..4+o) 202,1 W () 12862256 + 8 [0]\n\t\t#\n\t\tdir = $(6+o)\n\t\tloc = $(NF-3)\n\n\t\tif (starts[dev, loc] > 0) {\n\t\t\tlatency_ms = 1000 * (time - starts[dev, loc])\n\t\t\ti = 0\n\t\t\tfor (ms = 1; latency_ms > ms; ms *= 2) { i++ }\n\t\t\thist[i]++\n\t\t\tif (i > max_i)\n\t\t\t\tmax_i = i\n\t\t\tdelete starts[dev, loc]\n\t\t}\n\t\tnext\n\t}\n\n\t# timestamp\n\t$1 == \"time\" {\n\t\tlasttime = $2\n\t}\n\n\t# print summary\n\t$1 == \"tick\" {\n\t\tprint \"\"\n\t\tif (opt_timestamp)\n\t\t\tprint lasttime\n\n\t\t# find max value\n\t\tmax_v = 0\n\t\tfor (i = 0; i <= max_i; i++) {\n\t\t\tif (hist[i] > max_v)\n\t\t\t\tmax_v = hist[i]\n\t\t}\n\n\t\t# print histogram\n\t\tprintf \"%8s .. %-8s: %-8s |%-38s|\\n\", \">=(ms)\", \"<(ms)\",\n\t\t    \"I/O\", \"Distribution\"\n\t\tms = 1\n\t\tfrom = 0\n\t\tfor (i = 0; i <= max_i; i++) {\n\t\t\tprintf \"%8d -> %-8d: %-8d |%-38s|\\n\", from, ms,\n\t\t\t    hist[i], star(hist[i], max_v, 38)\n\t\t\tfrom = ms\n\t\t\tms *= 2\n\t\t}\n\t\tfflush()\n\t\tdelete hist\n\t\tdelete starts\t# invalid if events missed between snapshots\n\t\tmax_i = 0\n\t}\n\n\t$0 ~ /LOST.*EVENTS/ { print \"WARNING: \" $0 > \"/dev/stderr\" }\n'\n\n### end tracing\nend\n"
  },
  {
    "path": "iosnoop",
    "content": "#!/bin/bash\n#\n# iosnoop - trace block device I/O.\n#           Written using Linux ftrace.\n#\n# This traces disk I/O at the block device interface, using the block:\n# tracepoints. This can help characterize the I/O requested for the storage\n# devices and their resulting performance. I/O completions can also be studied\n# event-by-event for debugging disk and controller I/O scheduling issues.\n#\n# USAGE: ./iosnoop [-hQst] [-d device] [-i iotype] [-p pid] [-n name] [duration]\n#\n# Run \"iosnoop -h\" for full usage.\n#\n# REQUIREMENTS: FTRACE CONFIG, block:block_rq_* tracepoints (you may\n# already have these on recent kernels).\n#\n# OVERHEAD: By default, iosnoop works without buffering, printing I/O events\n# as they happen (uses trace_pipe), context switching and consuming CPU to do\n# so. This has a limit of about 10,000 IOPS (depending on your platform), at\n# which point iosnoop will be consuming 1 CPU. The duration mode uses buffering,\n# and can handle much higher IOPS rates, however, the buffer has a limit of\n# about 50,000 I/O, after which events will be dropped. You can tune this with\n# bufsize_kb, which is per-CPU. Also note that the \"-n\" option is currently\n# post-filtered, so all events are traced.\n#\n# This was written as a proof of concept for ftrace. It would be better written\n# using perf_events (after some capabilities are added), which has a better\n# buffering policy, or a tracer such as SystemTap or ktap.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the iosnoop(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 12-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock\nbufsize_kb=4096\nopt_duration=0; duration=; opt_name=0; name=; opt_pid=0; pid=; ftext=\nopt_start=0; opt_end=0; opt_device=0; device=; opt_iotype=0; iotype=\nopt_queue=0\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: iosnoop [-hQst] [-d device] [-i iotype] [-p PID] [-n name]\n\t               [duration]\n\t                 -d device       # device string (eg, \"202,1)\n\t                 -i iotype       # match type (eg, '*R*' for all reads)\n\t                 -n name         # process name to match on I/O issue\n\t                 -p PID          # PID to match on I/O issue\n\t                 -Q              # use queue insert as start time\n\t                 -s              # include start time of I/O (s)\n\t                 -t              # include completion time of I/O (s)\n\t                 -h              # this usage message\n\t                 duration        # duration seconds, and use buffers\n\t  eg,\n\t       iosnoop                   # watch block I/O live (unbuffered)\n\t       iosnoop 1                 # trace 1 sec (buffered)\n\t       iosnoop -Q                # include queueing time in LATms\n\t       iosnoop -ts               # include start and end timestamps\n\t       iosnoop -i '*R*'          # trace reads\n\t       iosnoop -p 91             # show I/O issued when PID 91 is on-CPU\n\t       iosnoop -Qp 91            # show I/O queued by PID 91, queue time\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > events/block/$b_start/enable\"\n\twarn \"echo 0 > events/block/block_rq_complete/enable\"\n\tif (( opt_device || opt_iotype || opt_pid )); then\n\t\twarn \"echo 0 > events/block/$b_start/filter\"\n\t\twarn \"echo 0 > events/block/block_rq_complete/filter\"\n\tfi\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts d:hi:n:p:Qst opt\ndo\n\tcase $opt in\n\td)\topt_device=1; device=$OPTARG ;;\n\ti)\topt_iotype=1; iotype=$OPTARG ;;\n\tn)\topt_name=1; name=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tQ)\topt_queue=1 ;;\n\ts)\topt_start=1 ;;\n\tt)\topt_end=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\nif (( $# )); then\n\topt_duration=1\n\tduration=$1\n\tshift\nfi\nif (( opt_device )); then\n\tmajor=${device%,*}\n\tminor=${device#*,}\n\tdev=$(( (major << 20) + minor ))\nfi\n\n### option logic\n(( opt_pid && opt_name )) && die \"ERROR: use either -p or -n.\"\n(( opt_pid )) && ftext=\" issued by PID $pid\"\n(( opt_name )) && ftext=\" issued by process name \\\"$name\\\"\"\nif (( opt_duration )); then\n\techo \"Tracing block I/O$ftext for $duration seconds (buffered)...\"\nelse\n\techo \"Tracing block I/O$ftext. Ctrl-C to end.\"\nfi\nif (( opt_queue )); then\n\tb_start=block_rq_insert\nelse\n\tb_start=block_rq_issue\nfi\n\n### select awk\n(( opt_duration )) && use=mawk || use=gawk\t# workaround for mawk fflush()\n[[ -x /usr/bin/$use ]] && awk=$use || awk=awk\nwroteflock=1\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\n\n### setup and begin tracing\necho nop > current_tracer\nwarn \"echo $bufsize_kb > buffer_size_kb\"\nfilter=\nif (( opt_iotype )); then\n\tfilter=\"rwbs ~ \\\"$iotype\\\"\"\nfi\nif (( opt_device )); then\n\t[[ \"$filter\" != \"\" ]] && filter=\"$filter && \"\n\tfilter=\"${filter}dev == $dev\"\nfi\nfilter_i=$filter\nif (( opt_pid )); then\n\t[[ \"$filter_i\" != \"\" ]] && filter_i=\"$filter_i && \"\n\tfilter_i=\"${filter_i}common_pid == $pid\"\n\t[[ \"$filter\" == \"\" ]] && filter=0\nfi\nif (( opt_iotype || opt_device || opt_pid )); then\n\tif ! echo \"$filter_i\" > events/block/$b_start/filter || \\\n\t    ! echo \"$filter\" > events/block/block_rq_complete/filter\n\tthen\n\t\tedie \"ERROR: setting -d or -t filter. Exiting.\"\n\tfi\nfi\nif ! echo 1 > events/block/$b_start/enable || \\\n    ! echo 1 > events/block/block_rq_complete/enable; then\n\tedie \"ERROR: enabling block I/O tracepoints. Exiting.\"\nfi\n(( opt_start )) && printf \"%-15s \" \"STARTs\"\n(( opt_end )) && printf \"%-15s \" \"ENDs\"\nprintf \"%-12.12s %-6s %-4s %-8s %-12s %-6s %8s\\n\" \\\n    \"COMM\" \"PID\" \"TYPE\" \"DEV\" \"BLOCK\" \"BYTES\" \"LATms\"\n\n#\n# Determine output format. It may be one of the following (newest first):\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#           TASK-PID    CPU#    TIMESTAMP  FUNCTION\n# To differentiate between them, the number of header fields is counted,\n# and an offset set, to skip the extra column when needed.\n#\noffset=$($awk 'BEGIN { o = 0; }\n\t$1 == \"#\" && $2 ~ /TASK/ && NF == 6 { o = 1; }\n\t$2 ~ /TASK/ { print o; exit }' trace)\n\n### print trace buffer\nwarn \"echo > trace\"\n( if (( opt_duration )); then\n\t# wait then dump buffer\n\tsleep $duration\n\tcat trace\nelse\n\t# print buffer live\n\tcat trace_pipe\nfi ) | $awk -v o=$offset -v opt_name=$opt_name -v name=$name \\\n    -v opt_duration=$opt_duration -v opt_start=$opt_start -v opt_end=$opt_end \\\n    -v b_start=$b_start '\n\t# common fields\n\t$1 != \"#\" {\n\t\t# task name can contain dashes\n\t\tcomm = pid = $1\n\t\tsub(/-[0-9][0-9]*/, \"\", comm)\n\t\tsub(/.*-/, \"\", pid)\n\t\ttime = $(3+o); sub(\":\", \"\", time)\n\t\tdev = $(5+o)\n\t}\n\n\t# block I/O request\n\t$1 != \"#\" && $0 ~ b_start {\n\t\tif (opt_name && match(comm, name) == 0)\n\t\t\tnext\n\t\t#\n\t\t# example: (fields1..4+o) 202,1 W 0 () 12862264 + 8 [tar]\n\t\t# The cmd field \"()\" might contain multiple words (hex),\n\t\t# hence stepping from the right (NF-3).\n\t\t#\n\t\tloc = $(NF-3)\n\t\tstarts[dev, loc] = time\n\t\tcomms[dev, loc] = comm\n\t\tpids[dev, loc] = pid\n\t\tnext\n\t}\n\n\t# block I/O completion\n\t$1 != \"#\" && $0 ~ /rq_complete/ {\n\t\t#\n\t\t# example: (fields1..4+o) 202,1 W () 12862256 + 8 [0]\n\t\t#\n\t\tdir = $(6+o)\n\t\tloc = $(NF-3)\n\t\tnsec = $(NF-1)\n\n\t\tif (starts[dev, loc] > 0) {\n\t\t\tlatency = sprintf(\"%.2f\",\n\t\t\t    1000 * (time - starts[dev, loc]))\n\t\t\tcomm = comms[dev, loc]\n\t\t\tpid = pids[dev, loc]\n\n\t\t\tif (opt_start)\n\t\t\t\tprintf \"%-15s \", starts[dev, loc]\n\t\t\tif (opt_end)\n\t\t\t\tprintf \"%-15s \", time\n\t\t\tprintf \"%-12.12s %-6s %-4s %-8s %-12s %-6s %8s\\n\",\n\t\t\t    comm, pid, dir, dev, loc, nsec * 512, latency\n\t\t\tif (!opt_duration)\n\t\t\t\tfflush()\n\n\t\t\tdelete starts[dev, loc]\n\t\t\tdelete comms[dev, loc]\n\t\t\tdelete pids[dev, loc]\n\t\t}\n\t\tnext\n\t}\n\n\t$0 ~ /LOST.*EVENTS/ { print \"WARNING: \" $0 > \"/dev/stderr\" }\n'\n\n### end tracing\nend\n"
  },
  {
    "path": "kernel/funccount",
    "content": "#!/bin/bash\n#\n# funccount - count kernel function calls matching specified wildcards.\n#             Uses Linux ftrace.\n#\n# This is a proof of concept using Linux ftrace capabilities on older kernels,\n# and works by using function profiling: in-kernel counters.\n#\n# USAGE: funccount [-hT] [-i secs] [-d secs] [-t top] funcstring\n#    eg,\n#        funccount 'ext3*'\t# count all ext3* kernel function calls\n#\n# Run \"funccount -h\" for full usage.\n#\n# WARNING: This uses dynamic tracing of kernel functions, and could cause\n# kernel panics or freezes. Test, and know what you are doing, before use.\n#\n# REQUIREMENTS: CONFIG_FUNCTION_PROFILER, awk.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 12-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nopt_duration=0; duration=; opt_interval=0; interval=999999; opt_timestamp=0\nopt_tail=0; tcmd=cat; ttext=\ntrap 'quit=1' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: funccount [-hT] [-i secs] [-d secs] [-t top] funcstring\n\t                 -d seconds      # total duration of trace\n\t                 -h              # this usage message\n\t                 -i seconds      # interval summary\n\t                 -t top          # show top num entries only\n\t                 -T              # include timestamp (for -i)\n\t  eg,\n\t       funccount 'vfs*'          # trace all funcs that match \"vfs*\"\n\t       funccount -d 5 'tcp*'     # trace \"tcp*\" funcs for 5 seconds\n\t       funccount -t 10 'ext3*'   # show top 10 \"ext3*\" funcs\n\t       funccount -i 1 'ext3*'    # summary every 1 second\n\t       funccount -i 1 -d 5 'ext3*' # 5 x 1 second summaries\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\n### process options\nwhile getopts d:hi:t:T opt\ndo\n\tcase $opt in\n\td)\topt_duration=1; duration=$OPTARG ;;\n\ti)\topt_interval=1; interval=$OPTARG ;;\n\tt)\topt_tail=1; tnum=$OPTARG ;;\n\tT)\topt_timestamp=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### option logic\n(( $# == 0 )) && usage\nfuncs=\"$1\"\nif (( opt_tail )); then\n\ttcmd=\"tail -$tnum\"\n\tttext=\" Top $tnum only.\"\nfi\nif (( opt_duration )); then\n\techo \"Tracing \\\"$funcs\\\" for $duration seconds.$ttext..\"\nelse\n\techo \"Tracing \\\"$funcs\\\".$ttext.. Ctrl-C to end.\"\nfi\n(( opt_duration && !opt_interval )) && interval=$duration\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### enable tracing\nsysctl -q kernel.ftrace_enabled=1\t# doesn't set exit status\necho \"$funcs\" > set_ftrace_filter || die \"ERROR: enabling \\\"$funcs\\\". Exiting.\"\nwarn \"echo nop > current_tracer\"\nif ! echo 1 > function_profile_enabled; then\n\techo > set_ftrace_filter\n\tdie \"ERROR: enabling function profiling.\"\\\n\t    \"Have CONFIG_FUNCTION_PROFILER? Exiting.\"\nfi\n\n### summarize\nquit=0; secs=0\nwhile (( !quit && (!opt_duration || secs < duration) )); do\n\t(( secs += interval ))\n\techo 0 > function_profile_enabled\n\techo 1 > function_profile_enabled\n\tsleep $interval\n\n\techo\n\t(( opt_timestamp )) && date\n\tprintf \"%-30s %8s\\n\" \"FUNC\" \"COUNT\"\n\n\tcat trace_stat/function* | awk '\n\t# skip headers by matching on the numeric hit column\n\t$2 ~ /[0-9]/ { a[$1] += $2 }\n\tEND {\n\t\tfor (k in a) {\n\t\t\tprintf \"%-30s %8d\\n\", k,  a[k]\n\t\t}\n\t}' | sort -n -k2 | $tcmd\ndone\n\n### end tracing\necho 2>/dev/null\necho \"Ending tracing...\" 2>/dev/null\nwarn \"echo 0 > function_profile_enabled\"\nwarn \"echo > set_ftrace_filter\"\n"
  },
  {
    "path": "kernel/funcgraph",
    "content": "#!/bin/bash\n#\n# funcgraph - trace kernel function graph, showing child function calls.\n#             Uses Linux ftrace.\n#\n# This is an exploratory tool that shows the graph of child function calls\n# for a given kernel function. This can cost moderate overhead to execute, and\n# should only be used to understand kernel behavior for a given function before\n# using other, lower overhead tools. This is a proof of concept using Linux\n# ftrace capabilities on older kernels.\n#\n# USAGE: funcgraph [-aCDhHPtT] [-m maxdepth] [-p PID] [-L TID] [-d secs] funcstring\n#\n# Run \"funcgraph -h\" for full usage.\n#\n# The output format is the same as the ftrace function graph trace format,\n# described in the kernel source under Documentation/trace/ftrace.txt.\n# Note that the output may be shuffled when different CPU buffers are read;\n# check the CPU column for changes, or include timestamps (-t) and post sort.\n#\n# The \"-d duration\" mode leaves the trace data in the kernel buffer, and\n# only reads it at the end. If the trace data is large, beware of exhausting\n# buffer space (/sys/kernel/debug/tracing/buffer_size_kb) and losing data.\n#\n# Also beware of feedback loops: tracing tcp* functions over an ssh session,\n# or writing ext4* functions to an ext4 file system. For the former, tcp\n# trace data could be redirected to a file (as in the usage message). For\n# the latter, trace to the screen or a different file system.\n#\n# WARNING: This uses dynamic tracing of kernel functions, and could cause\n# kernel panics or freezes. Test, and know what you are doing, before use.\n#\n# OVERHEADS: This tool causes moderate to high overheads. Use with caution for\n# exploratory purposes, then switch to lower overhead techniques based on\n# findings. It's expected that the kernel will run at least 50% slower while\n# this tool is running -- even while no output is being generated. This is\n# because ALL kernel functions are traced, and filtered based on the function\n# of interest. When output is generated, it can generate many lines quickly\n# depending on the traced event. Such data will cause performance overheads.\n# This also works without buffering by default, printing function events\n# as they happen (uses trace_pipe), context switching and consuming CPU to do\n# so. If needed, you can try the \"-d secs\" option, which buffers events\n# instead, reducing overhead. If you think the buffer option is losing events,\n# try increasing the buffer size (buffer_size_kb).\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 12-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock\nopt_duration=0; duration=; opt_pid=0; pid=; opt_tid=0; tid=; pidtext=\nopt_headers=0; opt_proc=0; opt_time=0; opt_tail=0; opt_nodur=0; opt_cpu=0\nopt_max=0; max=0\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: funcgraph [-aCDhHPtT] [-m maxdepth] [-p PID] [-L TID] [-d secs] funcstring\n\t                 -a              # all info (same as -HPt)\n\t                 -C              # measure on-CPU time only\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -D              # do not show function duration\n\t                 -h              # this usage message\n\t                 -H              # include column headers\n\t                 -m maxdepth     # max stack depth to show\n\t                 -p PID          # trace when this pid is on-CPU\n\t                 -L TID          # trace when this thread is on-CPU\n\t                 -P              # show process names & PIDs\n\t                 -t              # show timestamps\n\t                 -T              # comment function tails\n\t  eg,\n\t       funcgraph do_nanosleep    # trace do_nanosleep() and children\n\t       funcgraph -m 3 do_sys_open # trace do_sys_open() to 3 levels only\n\t       funcgraph -a do_sys_open    # include timestamps and process name\n\t       funcgraph -p 198 do_sys_open # trace vfs_read() for PID 198 only\n\t       funcgraph -d 1 do_sys_open >out # trace 1 sec, then write to file\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\n\t(( opt_time )) && warn \"echo nofuncgraph-abstime > trace_options\"\n\t(( opt_proc )) && warn \"echo nofuncgraph-proc > trace_options\"\n\t(( opt_tail )) && warn \"echo nofuncgraph-tail > trace_options\"\n\t(( opt_nodur )) && warn \"echo funcgraph-duration > trace_options\"\n\t(( opt_cpu )) && warn \"echo sleep-time > trace_options\"\n\n\twarn \"echo nop > current_tracer\"\n\t(( opt_pid || opt_tid )) && warn \"echo > set_ftrace_pid\"\n\t(( opt_max )) && warn \"echo 0 > max_graph_depth\"\n\twarn \"echo > set_graph_function\"\n\twarn \"echo > trace\"\n\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts aCd:DhHm:p:L:PtT opt\ndo\n\tcase $opt in\n\ta)\topt_headers=1; opt_proc=1; opt_time=1 ;;\n\tC)\topt_cpu=1; ;;\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tD)\topt_nodur=1; ;;\n\tm)\topt_max=1; max=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tL)\topt_tid=1; tid=$OPTARG ;;\n\tH)\topt_headers=1; ;;\n\tP)\topt_proc=1; ;;\n\tt)\topt_time=1; ;;\n\tT)\topt_tail=1; ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### option logic\n(( $# == 0 )) && usage\n(( opt_pid && opt_tid )) && edie \"ERROR: You can use -p or -L but not both.\"\nfuncs=\"$1\"\n(( opt_pid )) && pidtext=\" for PID $pid\"\n(( opt_tid )) && pidtext=\" for TID $tid\"\nif (( opt_duration )); then\n\techo \"Tracing \\\"$funcs\\\"$pidtext for $duration seconds...\"\nelse\n\techo \"Tracing \\\"$funcs\\\"$pidtext... Ctrl-C to end.\"\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and commence tracing\nsysctl -q kernel.ftrace_enabled=1\t# doesn't set exit status\nread mode < current_tracer\n[[ \"$mode\" != \"nop\" ]] && edie \"ERROR: ftrace active (current_tracer=$mode)\"\nif (( opt_max )); then\n\tif ! echo $max > max_graph_depth; then\n\t\tedie \"ERROR: setting -m $max. Older kernel version? Exiting.\"\n\tfi\nfi\nif (( opt_pid )); then\n    echo > set_ftrace_pid\n    # ftrace expects kernel pids, which are thread ids\n    for tid in /proc/$pid/task/*; do\n        if ! echo ${tid##*/} >> set_ftrace_pid; then\n            edie \"ERROR: setting -p $pid (PID exist?). Exiting.\"\n        fi\n    done\nfi\nif (( opt_tid )); then\n    if ! echo $tid > set_ftrace_pid; then\n        edie \"ERROR: setting -L $tid (TID exist?). Exiting.\"\n    fi\nfi\nif ! echo > set_ftrace_filter; then\n\tedie \"ERROR: writing to set_ftrace_filter. Exiting.\"\nfi\nif ! echo \"$funcs\" > set_graph_function; then\n\tedie \"ERROR: enabling \\\"$funcs\\\". Exiting.\"\nfi\nif ! echo function_graph > current_tracer; then\n\tedie \"ERROR: setting current_tracer to \\\"function\\\". Exiting.\"\nfi\nif (( opt_cpu )); then\n\tif ! echo nosleep-time > trace_options; then\n\t\tedie \"ERROR: setting -C (nosleep-time). Exiting.\"\n\tfi\nfi\n# the following must be done after setting current_tracer\nif (( opt_time )); then\n\tif ! echo funcgraph-abstime > trace_options; then\n\t\tedie \"ERROR: setting -t (funcgraph-abstime). Exiting.\"\n\tfi\nfi\nif (( opt_proc )); then\n\tif ! echo funcgraph-proc > trace_options; then\n\t\tedie \"ERROR: setting -P (funcgraph-proc). Exiting.\"\n\tfi\nfi\nif (( opt_tail )); then\n\tif ! echo funcgraph-tail > trace_options; then\n\t\tedie \"ERROR: setting -T (funcgraph-tail). Old kernel? Exiting.\"\n\tfi\nfi\nif (( opt_nodur )); then\n\tif ! echo nofuncgraph-duration > trace_options; then\n\t\tedie \"ERROR: setting -D (nofuncgraph-duration). Exiting.\"\n\tfi\nfi\n\n### print trace buffer\nwarn \"echo > trace\"\nif (( opt_duration )); then\n\tsleep $duration\n\tif (( opt_headers )); then\n\t\tcat trace\n\telse\n\t\tgrep -v '^#' trace\n\tfi\nelse\n\t# trace_pipe lack headers, so fetch them from trace\n\t(( opt_headers )) && cat trace\n\tcat trace_pipe\nfi\n\n### end tracing\nend\n"
  },
  {
    "path": "kernel/funcslower",
    "content": "#!/bin/bash\n#\n# funcslower - trace kernel functions slower than a threshold (microseconds).\n#              Uses Linux ftrace.\n#\n# This uses the Linux ftrace function graph profiler to time kernel functions\n# and filter them based on a latency threshold. This is a proof of concept using\n# Linux ftrace capabilities on older kernels.\n#\n# USAGE: funcslower [-aChHPt] [-p PID] [-d secs] funcstring latency_us\n#\n# Run \"funcslower -h\" for full usage.\n#\n# REQUIREMENTS: FTRACE function graph, which you may already have available\n# and enabled in recent kernels. And awk.\n#\n# The output format is the same as the ftrace function graph trace format,\n# described in the kernel source under Documentation/trace/ftrace.txt.\n# Note that the output may be shuffled when different CPU buffers are read;\n# check the CPU column for changes, or include timestamps (-t) and post sort.\n#\n# WARNING: This uses dynamic tracing of kernel functions, and could cause\n# kernel panics or freezes. Test, and know what you are doing, before use.\n#\n# OVERHEADS: Timing and filtering is performed in-kernel context, costing\n# lower overheads than post-processing in user space. If you trace frequent\n# events (eg, pick a common function and a low threshold), you might want to\n# try the \"-d secs\" option, which buffers events in-kernel instead of printing\n# them live.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 12-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock\nopt_duration=0; duration=; opt_pid=0; pid=; opt_tid=0; tid=\npidtext=; opt_headers=0; opt_proc=0; opt_time=0; opt_cpu=0\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: funcslower [-aChHPt] [-p PID] [-L TID] [-d secs] funcstring latency_us\n\t                 -a              # all info (same as -HPt)\n\t                 -C              # measure on-CPU time only\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -h              # this usage message\n\t                 -H              # include column headers\n\t                 -p PID          # trace when this pid is on-CPU\n\t                 -L TID          # trace when this thread is on-CPU\n\t                 -P              # show process names & PIDs\n\t                 -t              # show timestamps\n\t  eg,\n\t       funcslower vfs_read 10000 # trace vfs_read() slower than 10 ms\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\n\t(( opt_time )) && warn \"echo nofuncgraph-abstime > trace_options\"\n\t(( opt_proc )) && warn \"echo nofuncgraph-proc > trace_options\"\n\t(( opt_cpu )) && warn \"echo sleep-time > trace_options\"\n\n\twarn \"echo nop > current_tracer\"\n\t(( opt_pid )) && warn \"echo > set_ftrace_pid\"\n\twarn \"echo > set_ftrace_filter\"\n\twarn \"echo > set_graph_function\"\n\twarn \"echo 0 > tracing_thresh\"\n\twarn \"echo > trace\"\n\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts aCd:hHp:L:Pt opt\ndo\n\tcase $opt in\n\ta)\topt_headers=1; opt_proc=1; opt_time=1 ;;\n\tC)\topt_cpu=1; ;;\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n    L)  opt_tid=1; tid=$OPTARG ;;\n\tH)\topt_headers=1; ;;\n\tP)\topt_proc=1; ;;\n\tt)\topt_time=1; ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### option logic\n(( $# < 2 )) && usage\n(( opt_pid && opt_tid )) && edie \"ERROR: You can use -p or -L but not both.\"\nfuncs=\"$1\"\nshift\nthresh=$1\n(( opt_pid )) && pidtext=\" for PID $pid\"\n(( opt_tid )) && pidtext=\" for TID $tid\"\nprintf \"Tracing \\\"$funcs\\\"$pidtext slower than $thresh us\"\nif (( opt_duration )); then\n\techo \" for $duration seconds...\"\nelse\n\techo \"... Ctrl-C to end.\"\nfi\n\n## select awk\nif (( opt_duration )); then\n\t[[ -x /usr/bin/mawk ]] && awk=mawk || awk=awk\nelse\n\t# workarounds for mawk/gawk fflush behavior\n\tif [[ -x /usr/bin/gawk ]]; then\n\t\tawk=gawk\n\telif [[ -x /usr/bin/mawk ]]; then\n\t\tawk=\"mawk -W interactive\"\n\telse\n\t\tawk=awk\n\tfi\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and commence tracing\nsysctl -q kernel.ftrace_enabled=1\t# doesn't set exit status\nread mode < current_tracer\n[[ \"$mode\" != \"nop\" ]] && edie \"ERROR: ftrace active (current_tracer=$mode)\"\nif ! echo $thresh > tracing_thresh; then\n\tedie \"ERROR: setting tracing_thresh to $thresh. Exiting.\"\nfi\nif (( opt_pid )); then\n    echo '' > set_ftrace_pid\n    # ftrace expects kernel pids, which are thread ids\n    for tid in /proc/$pid/task/*; do\n        if ! echo ${tid##*/} >> set_ftrace_pid; then\n            edie \"ERROR: setting -p $pid (PID exist?). Exiting.\"\n        fi\n    done\nfi\nif (( opt_tid )); then\n    if ! echo $tid > set_ftrace_pid; then\n        edie \"ERROR: setting -L $tid (TID exist?). Exiting.\"\n    fi\nfi\nif ! echo \"$funcs\" > set_ftrace_filter; then\n\tedie \"ERROR: enabling \\\"$funcs\\\" filter. Function exist? Exiting.\"\nfi\nif ! echo \"$funcs\" > set_graph_function; then\n\tedie \"ERROR: enabling \\\"$funcs\\\" graph. Exiting.\"\nfi\nif ! echo function_graph > current_tracer; then\n\tedie \"ERROR: setting current_tracer to \\\"function_graph\\\". Exiting.\"\nfi\nif (( opt_cpu )); then\n\tif ! echo nosleep-time > trace_options; then\n\t\tedie \"ERROR: setting -C (nosleep-time). Exiting.\"\n\tfi\nfi\n# the following must be done after setting current_tracer\nif (( opt_time )); then\n\tif ! echo funcgraph-abstime > trace_options; then\n\t\tedie \"ERROR: setting -t (funcgraph-abstime). Exiting.\"\n\tfi\nfi\nif (( opt_proc )); then\n\tif ! echo funcgraph-proc > trace_options; then\n\t\tedie \"ERROR: setting -P (funcgraph-proc). Exiting.\"\n\tfi\nfi\n\n### setup output filter\ncat=cat\nif (( opt_proc )); then\n\t# remove proc change entries, since PID is included. example:\n\t#  ------------------------------------------\n\t#  0)  supervi-1699  =>  supervi-1693 \n\t#  ------------------------------------------\n\t#\n\tcat=$awk' \"/(^ ---|^$)/ || \\$3 == \\\"=>\\\" { next } { print \\$0 }\"'\nfi\n\n### print trace buffer\nwarn \"echo > trace\"\nif (( opt_duration )); then\n\tsleep $duration\n\tif (( opt_headers )); then\n\t\t$cat trace\n\telse\n\t\t$cat trace | grep -v '^#'\n\tfi\nelse\n\t# trace_pipe lack headers, so fetch them from trace\n\t(( opt_headers )) && cat trace\n\teval $cat trace_pipe\nfi\n\n### end tracing\nend\n"
  },
  {
    "path": "kernel/functrace",
    "content": "#!/bin/bash\n#\n# functrace - trace kernel function calls matching specified wildcards.\n#             Uses Linux ftrace.\n#\n# This is a proof of concept using Linux ftrace capabilities on older kernels.\n#\n# USAGE: functrace [-hH] [-p PID] [-L TID] [-d secs] funcstring\n#    eg,\n#        functrace '*sleep'\t# trace all functions ending in \"sleep\"\n#\n# Run \"functrace -h\" for full usage.\n#\n# The output format is the same as the ftrace function trace format, described\n# in the kernel source under Documentation/trace/ftrace.txt.\n#\n# The \"-d duration\" mode leaves the trace data in the kernel buffer, and\n# only reads it at the end. If the trace data is large, beware of exhausting\n# buffer space (/sys/kernel/debug/tracing/buffer_size_kb) and losing data.\n#\n# Also beware of feedback loops: tracing tcp* functions over an ssh session,\n# or writing ext4* functions to an ext4 file system. For the former, tcp\n# trace data could be redirected to a file (as in the usage message). For\n# the latter, trace to the screen or a different file system.\n#\n# WARNING: This uses dynamic tracing of kernel functions, and could cause\n# kernel panics or freezes. Test, and know what you are doing, before use.\n#\n# OVERHEADS: This can generate a lot of trace data quickly, depending on the\n# frequency of the traced events. Such data will cause performance overheads.\n# This also works without buffering by default, printing function events\n# as they happen (uses trace_pipe), context switching and consuming CPU to do\n# so. If needed, you can try the \"-d secs\" option, which buffers events\n# instead, reducing overhead. If you think the buffer option is losing events,\n# try increasing the buffer size (buffer_size_kb).\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 12-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock\nopt_duration=0; duration=; opt_pid=0; pid=; opt_tid=0; tid=; pidtext=\nopt_headers=0\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: functrace [-hH] [-p PID] [-L TID] [-d secs] funcstring\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -h              # this usage message\n\t                 -H              # include column headers\n\t                 -p PID          # trace when this pid is on-CPU\n\t                 -L TID          # trace when this thread is on-CPU\n\t  eg,\n\t       functrace do_nanosleep    # trace the do_nanosleep() function\n\t       functrace '*sleep'        # trace functions ending in \"sleep\"\n\t       functrace -p 198 'vfs*'   # trace \"vfs*\" funcs for PID 198\n\t       functrace 'tcp*' > out    # trace all \"tcp*\" funcs to out file\n\t       functrace -d 1 'tcp*' > out  # trace 1 sec, then write out file\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo nop > current_tracer\"\n\t(( opt_pid || opt_tid )) && warn \"echo > set_ftrace_pid\"\n\twarn \"echo > set_ftrace_filter\"\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts d:hHp:L: opt\ndo\n\tcase $opt in\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tL)\topt_tid=1; tid=$OPTARG ;;\n\tH)\topt_headers=1; ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### option logic\n(( $# == 0 )) && usage\n(( opt_pid && opt_tid )) && edie \"ERROR: You can use -p or -L but not both.\"\nfuncs=\"$1\"\n(( opt_pid )) && pidtext=\" for PID $pid\"\n(( opt_tid )) && pidtext=\" for TID $pid\"\nif (( opt_duration )); then\n\techo \"Tracing \\\"$funcs\\\"$pidtext for $duration seconds...\"\nelse\n\techo \"Tracing \\\"$funcs\\\"$pidtext... Ctrl-C to end.\"\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and commence tracing\nsysctl -q kernel.ftrace_enabled=1\t# doesn't set exit status\nread mode < current_tracer\n[[ \"$mode\" != \"nop\" ]] && edie \"ERROR: ftrace active (current_tracer=$mode)\"\nif (( opt_pid )); then\n    echo > set_ftrace_pid\n    # ftrace expects kernel pids, which are thread ids\n    for tid in /proc/$pid/task/*; do\n        if ! echo ${tid##*/} >> set_ftrace_pid; then\n            edie \"ERROR: setting -p $pid (PID exist?). Exiting.\"\n        fi\n    done\nfi\nif (( opt_tid )); then\n    if ! echo $tid > set_ftrace_pid; then\n        edie \"ERROR: setting -L $tid (TID exist?). Exiting.\"\n    fi\nfi\nif ! echo \"$funcs\" > set_ftrace_filter; then\n\tedie \"ERROR: enabling \\\"$funcs\\\". Exiting.\"\nfi\nif ! echo function > current_tracer; then\n\tedie \"ERROR: setting current_tracer to \\\"function\\\". Exiting.\"\nfi\n\n### print trace buffer\nwarn \"echo > trace\"\nif (( opt_duration )); then\n\tsleep $duration\n\tif (( opt_headers )); then\n\t\tcat trace\n\telse\n\t\tgrep -v '^#' trace\n\tfi\nelse\n\t# trace_pipe lack headers, so fetch them from trace\n\t(( opt_headers )) && cat trace\n\tcat trace_pipe\nfi\n\n### end tracing\nend\n"
  },
  {
    "path": "kernel/kprobe",
    "content": "#!/bin/bash\n#\n# kprobe - trace a given kprobe definition. Kernel dynamic tracing.\n#          Written using Linux ftrace.\n#\n# This will create, trace, then destroy a given kprobe definition. See\n# Documentation/trace/kprobetrace.txt in the Linux kernel source for the\n# syntax of a kprobe definition, and \"kprobe -h\" for examples. With this tool,\n# the probe alias is optional (it will become to kprobe:<funcname> if not\n# specified).\n#\n# USAGE: ./kprobe [-FhHsv] [-d secs] [-p pid] [-L tid] kprobe_definition [filter]\n#\n# Run \"kprobe -h\" for full usage.\n#\n# I wrote this because I kept testing different custom kprobes at the command\n# line, and wanted a way to automate the steps.\n#\n# WARNING: This uses dynamic tracing of kernel functions, and could cause\n# kernel panics or freezes, depending on the function traced. Test in a lab\n# environment, and know what you are doing, before use.\n#\n# REQUIREMENTS: FTRACE and KPROBE CONFIG, which you may already have on recent\n# kernel versions.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the kprobe(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 22-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock; wroteflock=0\nopt_duration=0; duration=; opt_pid=0; pid=; opt_tid=0; tid=\nopt_filter=0; filter=; opt_view=0; opt_headers=0; opt_stack=0; dmesg=2\ndebug=0; opt_force=0\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: kprobe [-FhHsv] [-d secs] [-p PID] [-L TID] kprobe_definition [filter]\n\t                 -F              # force. trace despite warnings.\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -p PID          # PID to match on events\n\t                 -L TID          # thread id to match on events\n\t                 -v              # view format file (don't trace)\n\t                 -H              # include column headers\n\t                 -s              # show kernel stack traces\n\t                 -h              # this usage message\n\t\n\tNote that these examples may need modification to match your kernel\n\tversion's function names and platform's register usage.\n\t   eg,\n\t       kprobe p:do_sys_open\n\t                                 # trace open() entry\n\t       kprobe r:do_sys_open\n\t                                 # trace open() return\n\t       kprobe 'r:do_sys_open \\$retval'\n\t                                 # trace open() return value\n\t       kprobe 'r:myopen do_sys_open \\$retval'\n\t                                 # use a custom probe name\n\t       kprobe 'p:myopen do_sys_open mode=%cx:u16'\n\t                                 # trace open() file mode\n\t       kprobe 'p:myopen do_sys_open filename=+0(%si):string'\n\t                                 # trace open() with filename\n\t       kprobe -s 'p:myprobe tcp_retransmit_skb'\n\t                                 # show kernel stacks\n\t       kprobe 'p:do_sys_open file=+0(%si):string' 'file ~ \"*stat\"'\n\t                                 # opened files ending in \"stat\"\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > events/kprobes/$kname/enable\"\n\tif (( opt_filter )); then\n\t\twarn \"echo 0 > events/kprobes/$kname/filter\"\n\tfi\n\twarn \"echo -:$kname >> kprobe_events\"\n\t(( opt_stack )) && warn \"echo 0 > options/stacktrace\"\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts Fd:hHp:L:sv opt\ndo\n\tcase $opt in\n\tF)\topt_force=1 ;;\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tL)\topt_tid=1; tid=$OPTARG ;;\n\tH)\topt_headers=1 ;;\n\ts)\topt_stack=1 ;;\n\tv)\topt_view=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n(( $# )) || usage\nkprobe=$1\nshift\nif (( $# )); then\n\topt_filter=1\n\tfilter=$1\nfi\n\n### option logic\n(( opt_pid + opt_filter + opt_tid > 1 )) && \\\n\tdie \"ERROR: use at most one of -p, -L, or filter.\"\n(( opt_duration && opt_view )) && die \"ERROR: use either -d or -v.\"\nif (( opt_pid )); then\n\t# convert to filter\n\topt_filter=1\n\t# ftrace common_pid is thread id from user's perspective\n\tfor tid in /proc/$pid/task/*; do\n\t\tfilter=\"$filter || common_pid == ${tid##*/}\"\n\tdone\n\tfilter=${filter:3}  # trim leading ' || ' (four characters)\nfi\nif (( opt_tid )); then\n\topt_filter=1\n\tfilter=\"common_pid == $tid\"\nfi\nif [[ \"$kprobe\" != p:* && \"$kprobe\" != r:* ]]; then\n\techo >&2 \"ERROR: invalid kprobe definition (should start with p: or r:)\"\n\tusage\nfi\n#\n# parse the following:\n# r:do_sys_open\n# r:my_sys_open do_sys_open\n# r:do_sys_open %ax\n# r:do_sys_open $retval %ax\n# r:my_sys_open do_sys_open $retval %ax\n# r:do_sys_open rval=$retval\n# r:my_sys_open do_sys_open rval=$retval\n# r:my_sys_open do_sys_open rval=$retval %ax\n# ... and examples from USAGE message\n#\nkrest=${kprobe#*:}\nkname=${krest%% *}\nset -- $krest\nif [[ $2 == \"\" || $2 == *[=%\\$]* ]]; then\n\t# if probe name unspecified, default to function name\n\tktype=${kprobe%%:*}\n\tkprobe=\"$ktype:$kname $krest\"\nfi\nif (( debug )); then\n\techo \"kname: $kname, kprobe: $kprobe\"\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n## check function\nset -- $kprobe\nfname=$2\nif (( !opt_force )) && ! grep -w $fname available_filter_functions >/dev/null \\\n    2>&1\nthen\n\techo >&2 \"ERROR: func $fname not in $PWD/available_filter_functions.\"\n\tprintf >&2 \"Either it doesn't exist, or, it might be unsafe to kprobe. \"\n\techo >&2 \"Exiting. Use -F to override.\"\n\texit 1\nfi\n\nif (( !opt_view )); then\n\tif (( opt_duration )); then\n\t\techo \"Tracing kprobe $kname for $duration seconds (buffered)...\"\n\telse\n\t\techo \"Tracing kprobe $kname. Ctrl-C to end.\"\n\tfi\nfi\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and begin tracing\necho nop > current_tracer\nif ! echo \"$kprobe\" >> kprobe_events; then\n\techo >&2 \"ERROR: adding kprobe \\\"$kprobe\\\".\"\n\tif (( dmesg )); then\n\t\techo >&2 \"Last $dmesg dmesg entries (might contain reason):\"\n\t\tdmesg | tail -$dmesg | sed 's/^/    /'\n\tfi\n\tedie \"Exiting.\"\nfi\nif (( opt_view )); then\n\tcat events/kprobes/$kname/format\n\tedie \"\"\nfi\nif (( opt_filter )); then\n\tif ! echo \"$filter\" > events/kprobes/$kname/filter; then\n\t\tedie \"ERROR: setting filter or -p. Exiting.\"\n\tfi\nfi\nif (( opt_stack )); then\n\tif ! echo 1 > options/stacktrace; then\n\t\tedie \"ERROR: enabling stack traces (-s). Exiting\"\n\tfi\nfi\nif ! echo 1 > events/kprobes/$kname/enable; then\n\tedie \"ERROR: enabling kprobe $kname. Exiting.\"\nfi\n\n### print trace buffer\nwarn \"echo > trace\"\nif (( opt_duration )); then\n\tsleep $duration\n\tif (( opt_headers )); then\n\t\tcat trace\n\telse\n\t\tgrep -v '^#' trace\n\tfi\nelse\n\t# trace_pipe lack headers, so fetch them from trace\n\t(( opt_headers )) && cat trace\n\tcat trace_pipe\nfi\n\n### end tracing\nend\n"
  },
  {
    "path": "killsnoop",
    "content": "#!/bin/bash\n#\n# killsnoop - trace kill() syscalls with signal/process details.\n#             Written using Linux ftrace.\n#\n# This traces kill() syscalls, showing which process killed which pid and\n# returns the returncode (0 for success, -1 for error).\n#\n# This implementation is designed to work on older kernel versions, and without\n# kernel debuginfo. It works by dynamic tracing of the return value of kill()\n# and associating it with the previous kill() syscall return.\n# This approach is kernel version specific, and may not work on your version.\n# It is a workaround, and proof of concept for ftrace, until more kernel tracing\n# functionality is available.\n#\n# USAGE: ./killsnoop [-hst] [-d secs] [-p pid] [-n name]\n#\n# Run \"killsnoop -h\" for full usage.\n#\n# REQUIREMENTS: FTRACE and KPROBE CONFIG, syscalls:sys_enter_kill and\n# syscalls:sys_exit_kill kernel tracepoints (you may already have these\n# on recent kernels) and awk.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the killsnoop(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n# COPYRIGHT: Copyright (c) 2014 Martin Probst.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 20-Jul-2014   Brendan Gregg   Templated this.\n# 13-Sep-2014   Martin Probst   Created this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock; wroteflock=0\nopt_duration=0; duration=; opt_name=0; name=; opt_pid=0; pid=; ftext=\nopt_time=0; opt_fail=0; opt_file=0; file=\nkevent_entry=events/syscalls/sys_enter_kill\nkevent_return=events/syscalls/sys_exit_kill\ntrap ':' INT QUIT TERM PIPE HUP # sends execution to end tracing section\n\nfunction usage {\n    cat <<-END >&2\nUSAGE: killsnoop [-hst] [-d secs] [-p PID] [-n name] [filename]\n                 -d seconds      # trace duration, and use buffers\n                 -n name         # process name to match \n                 -p PID          # PID to match on kill issue\n                 -t              # include time (seconds)\n                 -s              # human readable signal names\n                 -h              # this usage message\n  eg,\n       killsnoop                 # watch kill()s live (unbuffered)\n       killsnoop -d 1            # trace 1 sec (buffered)\n       killsnoop -p 181          # trace kill()s issued to PID 181 only\n\nSee the man page and example file for more info.\nEND\n    exit\n}\n\nfunction warn {\n    if ! eval \"$@\"; then\n        echo >&2 \"WARNING: command failed \\\"$@\\\"\"\n    fi\n}\n\nfunction end {\n    # disable tracing\n    echo 2>/dev/null\n    echo \"Ending tracing...\" 2>/dev/null\n    cd $tracing\n    warn \"echo 0 > $kevent_entry/enable\"\n    warn \"echo 0 > $kevent_return/enable\"\n    warn \"echo > trace\"\n    (( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n    echo >&2 \"$@\"\n    exit 1\n}\n\nfunction edie {\n    # die with a quiet end()\n    echo >&2 \"$@\"\n    exec >/dev/null 2>&1\n    end\n    exit 1\n}\n\n### process options\nwhile getopts d:hn:p:st opt\ndo\n    case $opt in\n    d)  opt_duration=1; duration=$OPTARG ;;\n    n)  opt_name=1; name=$OPTARG ;;\n    p)  opt_pid=1; pid=$OPTARG ;;\n    t)  opt_time=1 ;;\n    s)  opt_fancy=1 ;;\n    h|?)    usage ;;\n    esac\ndone\nshift $(( $OPTIND - 1 ))\n(( $# )) && usage\n\n### option logic\n(( opt_pid && opt_name )) && die \"ERROR: use either -p or -n.\"\n(( opt_pid )) && ftext=\" issued to PID $pid\"\n(( opt_name )) && ftext=\" issued by process name \\\"$name\\\"\"\nif (( opt_duration )); then\n    echo \"Tracing kill()s$ftext for $duration seconds (buffered)...\"\nelse\n    echo \"Tracing kill()s$ftext. Ctrl-C to end.\"\nfi\n\n### select awk\n# workaround for mawk fflush()\n[[ -x /usr/bin/mawk ]] && awk=\"mawk\" && mawk -W interactive && \\\n[ $? -eq 0 ] && awk=\"mawk -W interactive\"\n# workaround for gawk strtonum()\n[[ -x /usr/bin/gawk ]] && awk=\"gawk --non-decimal-data\"\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and begin tracing\necho nop > current_tracer\nif ! echo 1 > $kevent_entry/enable; then\n    edie \"ERROR: enabling kill() entry tracepoint Exiting.\"\nfi\nif ! echo 1 > $kevent_return/enable; then\n    edie \"ERROR: enabling kill() return tracepoint. Exiting.\"\nfi\n(( opt_time )) && printf \"%-16s \" \"TIMEs\"\nprintf \"%-16.16s %-6s %-8s %-10s %4s\\n\" \"COMM\" \"PID\" \"TPID\" \"SIGNAL\" \"RETURN\"\n\n#\n# Determine output format. It may be one of the following (newest first):\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#           TASK-PID    CPU#    TIMESTAMP  FUNCTION\n# To differentiate between them, the number of header fields is counted,\n# and an offset set, to skip the extra column when needed.\n#\noffset=$($awk 'BEGIN { o = 0; }\n    $1 == \"#\" && $2 ~ /TASK/ && NF == 6 { o = 1; }\n    $2 ~ /TASK/ { print o; exit }' trace)\n\n### print trace buffer\nwarn \"echo > trace\"\n( if (( opt_duration )); then\n    # wait then dump buffer\n    sleep $duration\n    cat trace\nelse\n    # print buffer live\n    cat trace_pipe\nfi ) | $awk -v o=$offset -v opt_name=$opt_name -v name=$name \\\n    -v opt_duration=$opt_duration -v opt_time=$opt_time \\\n    -v opt_pid=$pid -v opt_fancy=$opt_fancy '\n    # fancy signal names\n    BEGIN {\n        signals[1] = \"SIGHUP\"\n        signals[2] = \"SIGINT\"\n        signals[3] = \"SIGQUIT\"\n        signals[4] = \"SIGILL\"\n        signals[6] = \"SIGABRT\"\n        signals[8] = \"SIGFPE\"\n        signals[9] = \"SIGKILL\"\n        signals[11] = \"SIGSEGV\"\n        signals[13] = \"SIGPIPE\"\n        signals[14] = \"SIGALRM\"\n        signals[15] = \"SIGTERM\"\n        signals[10] = \"SIGUSR1\"\n        signals[12] = \"SIGUSR2\"\n        signals[17] = \"SIGCHLD\"\n        signals[18] = \"SIGCONT\"\n        signals[19] = \"SIGSTOP\"\n        signals[20] = \"SIGTSTP\"\n        signals[21] = \"SIGTTIN\"\n        signals[22] = \"SIGTTOU\"\n    }\n\n    # common fields\n    $1 != \"#\" {\n        # task name can contain dashes\n        comm = pid = $1\n        sub(/-[0-9][0-9]*/, \"\", comm)\n        if (opt_name && match(comm, name) == 0)\n            next\n        sub(/.*-/, \"\", pid)\n    }\n\n    # sys_kill() entry\n    $1 != \"#\" && $(4+o) ~ /sys_kill/ && $(5+o) !~ /->/ {\n        #\n        # eg: ... sys_kill(pid:...\n        #\n        kpid = $(5+o)\n        signal = $(7+o)\n        sub(/,$/, \"\", kpid)\n        sub(/\\)$/, \"\", signal)\n        kpid = int(\"0x\"kpid)\n        signal = int(\"0x\"signal)\n        current[pid,\"kpid\"] = kpid\n        current[pid,\"signal\"] = signal\n    }\n\n    # sys_kill exit\n    $1 != \"#\" && $(5+o) ~ /->/ {\n        rv = int($NF)\n        killed_pid = current[pid,\"kpid\"]\n        signal = current[pid,\"signal\"]\n\n        delete current[pid,\"kpid\"]\n        delete current[pid,\"signal\"]\n\n        if(opt_pid && killed_pid != opt_pid) {\n            next\n        }\n\n        if (opt_time) {\n            time = $(3+o); sub(\":\", \"\", time)\n            printf \"%-16s \", time\n        }\n\n        if (opt_fancy) {\n            if (signals[signal] != \"\") {\n                signal = signals[signal]\n            }\n        }\n\n        printf \"%-16.16s %-6s %-8s %-10s %-4s\\n\", comm, pid, killed_pid, signal,\n            rv\n    }\n\n    $0 ~ /LOST.*EVENTS/ { print \"WARNING: \" $0 > \"/dev/stderr\" }\n'\n\n### end tracing\nend\n"
  },
  {
    "path": "man/man8/bitesize.8",
    "content": ".TH bitesize 8  \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nbitesize \\- show disk I/O size as a histogram. Uses Linux perf_events.\n.SH SYNOPSIS\n.B bitesize\n[-h] [-b buckets] [seconds]\n.SH DESCRIPTION\nThis can be used to characterize the distribution of block device (disk) I/O\nsizes. To study block device I/O in more detail, see iosnoop(8).\n\nThis uses multiple counting tracepoints with different filters, one for each\nhistogram bucket. While this is summarized in-kernel, the use of multiple\ntracepoints does add addiitonal overhead, which is more evident if you add\nmore buckets. In the future this functionality will be available in an\nefficient way in the kernel, and this tool can be rewritten.\n.SH REQUIREMENTS\nLinux perf_events: add linux-tools-common, run \"perf\", then add any additional\npackages it requests. This also requires the block:block_rq_issue tracepoint,\nwhich should already be available in recent kernels.\n.SH OPTIONS\n.TP\n\\-h\nUsage message.\n.TP\n\\-b buckets\nSpecify a list of bucket points for the histogram as a string (eg, \"10 500\n1000\"). The histogram will include buckets for less-than the minimum, and\ngreater-than-or-equal-to the maximum.  If a single value is specified, two\nstatistics only are gathered: for less-than and for greater-than-or-equal-to.\nThe overhead is relative to the number of buckets, so only specifying a\nsingle value costs the lowest overhead.\n.TP\nseconds\nNumber of seconds to trace. If not specified, this runs until Ctrl-C.\n.SH EXAMPLES\n.TP\nTrace read() syscalls until Ctrl-C, and show histogram of requested size:\n#\n.B bitesize syscalls:sys_enter_read count\n.SH FIELDS\n.TP\nKbytes\nKbyte range of the histogram bucket.\n.TP\nI/O\nNumber of I/O that occurred in this range while tracing.\n.TP\nDistribution\nASCII histogram representation of the I/O column.\n.SH OVERHEAD\nWhile the counts are performed in-kernel, there is one tracepoint used per\nhistogram bucket, so the overheads are higher than usual (relative to the\nnumber of buckets) than function counting using perf stat. The lowest\noverhead is when \\-b is used to specify one bucket only, bifurcating\nstatistics.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\niosnoop(8), iolatency(8), iostat(1)\n"
  },
  {
    "path": "man/man8/cachestat.8",
    "content": ".TH cachestat 8  \"2014-12-28\" \"USER COMMANDS\"\n.SH NAME\ncachestat \\- Measure page cache hits/misses. Uses Linux ftrace.\n.SH SYNOPSIS\n.B cachestat\n[\\-Dht] [interval]\n.SH DESCRIPTION\nThis tool provides basic cache hit/miss statistics for the Linux page cache.\n\nIts current implementation uses Linux ftrace dynamic function profiling to\ncreate custom in-kernel counters, which is a workaround until such counters\ncan be built-in to the kernel. Specifically, four kernel functions are counted:\n.IP\nmark_page_accessed() for measuring cache accesses\n.IP\nmark_buffer_dirty() for measuring cache writes\n.IP\nadd_to_page_cache_lru() for measuring page additions\n.IP\naccount_page_dirtied() for measuring page dirties\n.PP\nIt is possible that these functions have been renamed (or are different\nlogically) for your kernel version, and this script will not work as-is.\nThis was written for a Linux 3.13 kernel, and tested on a few others versions.\nThis script is a sandcastle: the kernel may wash some away, and you'll\nneed to rebuild.\n\nThis program's implementation can be improved in the future when other\nkernel capabilities are made available. If you need a more reliable tool now,\nthen consider other tracing alternatives (eg, SystemTap). This tool is really\na proof of concept to see what ftrace can currently do.\n\nWARNING: This uses dynamic tracing of kernel functions, and could cause\nkernel panics or freezes. Test, and know what you are doing, before use.\nIt also traces cache activity, which can be frequent, and cost some overhead.\nThe statistics should be treated as best-effort: there may be some error\nmargin depending on unusual workload types.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nCONFIG_FUNCTION_PROFILER, which you may already have enabled and available on\nrecent kernels, and awk.\n.SH OPTIONS\n.TP\n\\-D\nInclude extra fields for debug purposes (see script).\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-t\nInclude timestamps in units of seconds.\n.TP\ninterval\nOutput interval in seconds. Default is 1.\n.SH EXAMPLES\n.TP\nShow per-second page cache statistics:\n#\n.B cachestat\n.SH FIELDS\n.TP\nTIME\nTime, in HH:MM:SS.\n.TP\nHITS\nNumber of page cache hits (reads). Each hit is for one memory page (the size\ndepends on your processor architecture; commonly 4 Kbytes). Since this tool\noutputs at a timed interval, this field indicates the cache hit rate.\n.TP\nMISSES\nNumber of page cache misses (reads from storage I/O). Each miss is for one\nmemory page. Cache misses should be causing disk I/O. Run iostat(1) for\ncorrelation (although the miss count and size by the time disk I/O is issued\ncan differ due to I/O subsystem merging).\n.TP\nDIRTIES\nNumber of times a page in the page cache was written to and thus \"dirtied\".\nThe same page may be counted multiple times per interval, if it is written\nto multiple times. This field gives an indication of how much cache churn there\nis, caused by applications writing data.\n.TP\nRATIO\nThe ratio of cache hits to total cache accesses (hits + misses), as a\npercentage.\n.TP\nBUFFERS_MB\nSize of the buffer cache, for disk I/O. From /proc/meminfo.\n.TP\nCACHED_MB\nSize of the page cache, for file system I/O. From /proc/meminfo.\n.SH OVERHEAD\nThis tool currently uses ftrace function profiling, which provides efficient\nin-kernel counters. However, the functions profiled are executed frequently,\nso the overheads can add up. Test and measure before use. My own testing\nshowed around a 2% loss in application performance while this tool was running.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\niostat(1), iosnoop(8)\n"
  },
  {
    "path": "man/man8/execsnoop.8",
    "content": ".TH execsnoop 8  \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nexecsnoop \\- trace process exec() with arguments. Uses Linux ftrace.\n.SH SYNOPSIS\n.B execsnoop\n[\\-hrt] [\\-a argc] [\\-d secs] [name]\n.SH DESCRIPTION\nexecsnoop traces process execution, showing PID, PPID, and argument details\nif possible.\n\nThis traces exec() from the fork()->exec() sequence, which means it won't\ncatch new processes that only fork(). With the -r option, it will also catch\nprocesses that re-exec. It makes a best-effort attempt to retrieve the program\narguments and PPID; if these are unavailable, 0 and \"[?]\" are printed\nrespectively. There is also a limit to the number of arguments printed (by\ndefault, 8), which can be increased using -a.\n\nThis implementation is designed to work on older kernel versions, and without\nkernel debuginfo. It works by dynamic tracing an execve kernel function to\nread the arguments from the %si register. The stub_execve() function is tried\nfirst, and then the do_execve() function. The sched:sched_process_fork\ntracepoint, is used for the PPID. Tracing registers and kernel functions is\nan unstable technique, and this tool may not work for some kernels or platforms.\n\nThis program is a workaround that should be\nimproved in the future when other kernel capabilities are made available. If\nyou need a more reliable tool now, then consider other tracing alternatives\n(eg, SystemTap). This tool is really a proof of concept to see what ftrace can\ncurrently do.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE and KPROBE CONFIG, sched:sched_process_fork tracepoint,\nand either the stub_execve() or do_execve() kernel function. You may already\nhave these on recent kernels. And awk.\n.SH OPTIONS\n.TP\n\\-a argc\nMaximum number of arguments to show. The default is 8, and the maximum allowed\nis 16. If execsnoop thinks it has truncated the argument list, an ellipsis \n\"[...]\" will be shown.\n.TP\n\\-d seconds\nDuration to trace, in seconds. This also uses in-kernel buffering.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-r\nInclude re-exec()s.\n.TP\n\\-t\nInclude timestamps in units of seconds.\n.TP\nname\nOnly show processes that match this name.\nPartials and regular expressions are allowed, as this is filtered in\nuser space by awk.\n.SH EXAMPLES\n.TP\nTrace all new processes and arguments (if possible):\n#\n.B execsnoop\n.TP\nTrace all new process names containing the text \"http\":\n#\n.B execsnoop http\n.SH FIELDS\n.TP\nTIMEs\nTime of the exec(), in seconds.\n.TP\nPID\nProcess ID.\n.TP\nPPID\nParent process ID, if this was able to be read. If it wasn't, 0 is printed.\n.TP\nARGS\nCommand line arguments, if these were able to be read. If they aren't able to be\nread, \"[?]\" is printed (which would be due to a limitation in this tools\nimplementation, since this is workaround for older kernels; if you need\nreliable argument tracing, use a different tracer). They will be truncated\nto the argc limit, and an ellipsis \"[...]\" may be printed if execsnoop is\naware of the truncation.\n.SH OVERHEAD\nThis reads and processes exec() events in user space as they occur. Since the\nrate of exec() is expected to be low (< 500/s), the overhead is expected to\nbe small or negligible.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\ntop(1)\n"
  },
  {
    "path": "man/man8/funccount.8",
    "content": ".TH funccount 8  \"2014-07-19\" \"USER COMMANDS\"\n.SH NAME\nfunccount \\- count kernel function calls matching specified wildcards. Uses Linux ftrace.\n.SH SYNOPSIS\n.B funccount\n[\\-hT] [\\-i secs] [\\-d secs] [\\-t top] funcstring\n.SH DESCRIPTION\nThis tool is a quick way to determine which kernel functions are being called,\nand at what rate. It uses ftrace function profiling capabilities.\n\nWARNING: This uses dynamic tracing of (what can be many) kernel functions,\nand could cause kernel panics or freezes. Test, and know what you are doing,\nbefore use.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nCONFIG_FUNCTION_PROFILER, which you may already have enabled and available on\nrecent kernels, and awk.\n.SH OPTIONS\n\\-d seconds\nTotal duration of the trace.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-i seconds\nPrint an interval summary every so many seconds.\n.TP\n\\-t top\nPrint top number of entries only.\n.TP\n\\-T\nInclude timestamp on each summary.\n.TP\nfuncstring\nA function name to trace, which may include file glob style wildcards (\"*\") at\nthe beginning or ending of a string only. Eg, \"vfs*\" means match \"vfs\" followed\nby anything.\n.SH EXAMPLES\n.TP\nCount every kernel function beginning with \"bio_\", until Ctrl-C is hit:\n#\n.B funccount 'bio_*'\n.TP\nCount every \"tcp_*\" kernel function, and print a summary every one second, five in total:\n#\n.B funccount \\-i 1 \\-d 5 'tcp_*'\n.TP\nCount every \"ext4*\" kernel function, and print the top 20 when Ctrl-C is hit:\n#\n.B funccount \\-t 20 'ext4*'\n.SH FIELDS\n.TP\nFUNC\nKernel function name.\n.TP\nCOUNT\nNumber of times this function was called during the tracing interval.\n.SH OVERHEAD\nThis uses the ftrace profiling framework, which does in-kernel counts,\nlowering the overhead (compared to tracing each event).\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nfunctrace(8)\n"
  },
  {
    "path": "man/man8/funcgraph.8",
    "content": ".TH funcgraph 8  \"2014-07-29\" \"USER COMMANDS\"\n.SH NAME\nfuncgraph \\- trace kernel function graph, showing child function calls and times. Uses Linux ftrace.\n.SH SYNOPSIS\n.B funcgraph\n[\\-aCDhHPtT] [\\-m maxdepth] [\\-p PID] [\\-L TID] [\\-d secs] funcstring\n.SH DESCRIPTION\nThis is an exploratory tool that shows the graph of child function calls\nfor a given kernel function. This can cost moderate overhead to execute, and\nshould only be used to understand kernel behavior before using other, lower\noverhead tools. This is a proof of concept using Linux ftrace capabilities\non older kernels.\n\nThe output format is the same as the ftrace function graph trace format,\ndescribed in the kernel source under Documentation/trace/ftrace.txt.\nNote that the output may be shuffled when different CPU buffers are read;\ncheck the CPU column for changes, or include timestamps (-t) and post sort.\n\nThe \"-d duration\" mode leaves the trace data in the kernel buffer, and\nonly reads it at the end. If the trace data is large, beware of exhausting\nbuffer space (/sys/kernel/debug/tracing/buffer_size_kb) and losing data.\n\nAlso beware of feedback loops: tracing tcp* functions over an ssh session,\nor writing ext4* functions to an ext4 file system. For the former, tcp\ntrace data could be redirected to a file (as in the usage message). For\nthe latter, trace to the screen or a different file system.\n\nWARNING: This uses dynamic tracing of kernel functions, and could cause\nkernel panics or freezes. Test, and know what you are doing, before use.\nAlso see the OVERHEAD section.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE CONFIG, which you may already have enabled and available on recent\nkernels.\n.SH OPTIONS\n.TP\n\\-a\nAll info. Same as \\-HPt. (But no -T, which isn't available in older kernels.)\n.TP\n\\-C\nFunction durations measure on-CPU time only (exclude sleep time).\n.TP\n\\-d seconds\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-D\nDo not show function duration times.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-H\nPrint column headers.\n.TP\n\\-m\nMax depth to trace functions. By default, unlimited (0). This feature is only\navailable for newer Linux kernel versions.\n.TP\n\\-p PID\nOnly trace kernel functions when this process ID is on-CPU.\n.TP\n\\-L TID\nOnly trace kernel functions when this thread ID is on-CPU.\n.TP\n\\-P\nShow process names and process IDs with every line of output.\n.TP\n\\-t\nShow timestamps on every line of output.\n.TP\n\\-T\nTail mode: decorate function return lines with the name of the function. This\noption may not be available for older kernels.\n.TP\nfuncstring\nA function name to trace, which may include file glob style wildcards (\"*\") at\nthe beginning or ending of a string only. Eg, \"vfs*\" means match \"vfs\" followed\nby anything. Since the output is verbose, you probably only want to trace\nsingle functions, and not use wildcards.\n.SH EXAMPLES\n.TP\nTrace calls to do_nanosleep(), showing child functions and durations:\n#\n.B funcgraph do_nanosleep\n.TP\nSame as above, but include column headers:\n#\n.B funcgraph -H do_nanosleep\n.TP\nSame as above, but include timestamps and process names as well:\n#\n.B funcgraph -HtP do_nanosleep\n.TP\nTrace all vfs_read() kernel function calls, and child functions, for PID 198 only:\n#\n.B funcgraph \\-p 198 vfs_read\n.TP\nTrace all vfs_read() kernel function calls, and child functions, for 1 second then write to a file.\n#\n.B funcgraph \\-d 1 vfs_read > out\n.SH FIELDS\nThe output format depends on the kernel version, and headings can be printed\nusing \\-H. The format is the same as the ftrace function trace format, described\nin the kernel source under Documentation/trace/ftrace.txt.\n\nTypical fields are:\n.TP\nTIME\n(Shown with \\-t.) Time of event, in seconds.\n.TP\nCPU\nThe CPU this event occurred on.\n.TP\nTASK/PID\n(Shown with \\-P.) The process name (which could include dashes), a dash, and the process ID.\n.TP\nDURATION\nElapsed time during the function call, inclusive of children. This is also\ninclusive of sleep time, unless -C is used. The time is either displayed on\nthe return of a function (\"}\"), or for a leaf function (no children), on the\nsame line.\n\nIf the trace output begins with some returns that lack entries, their durations\nmay not be trusted. This is usually only the case for the first dozen or so\nlines.\n.TP\nFUNCTION CALLS\nEntries and returns from kernel functions.\n.SH OVERHEAD\nThis tool causes moderate to high overheads. Use with caution for\nexploratory purposes, then switch to lower overhead techniques based on\nfindings. It's expected that the kernel will run at least 50% slower while\nthis tool is running -- even while no output is being generated. This is\nbecause ALL kernel functions are traced, and filtered based on the function\nof interest. When output is generated, it can generate many lines quickly\ndepending on the traced event. Such data will cause performance overheads.\nThis also works without buffering by default, printing function events\nas they happen (uses trace_pipe), context switching and consuming CPU to do\nso. If needed, you can try the \"-d secs\" option, which buffers events\ninstead, reducing overhead. If you think the buffer option is losing events,\ntry increasing the buffer size (buffer_size_kb).\n\nIt's a good idea to use funccount(8) first, which is lower overhead, to\nhelp you select which functions you may want to trace using funcgraph(8).\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nfunccount(8), functrace(8), kprobe(8)\n"
  },
  {
    "path": "man/man8/funcslower.8",
    "content": ".TH funcslower 8  \"2014-07-30\" \"USER COMMANDS\"\n.SH NAME\nfuncslower \\- trace kernel functions slower than a threshold (microseconds). Uses Linux ftrace.\n.SH SYNOPSIS\n.B funcslower\n[\\-aChHPt] [\\-p PID] [\\-L TID] [\\-d secs] funcstring latency_us\n.SH DESCRIPTION\nThis uses the Linux ftrace function graph profiler to time kernel functions\nand filter them based on a latency threshold. Latency outliers can be studied\nthis way, confirming their presence, duration, and rate. This tool\nis a proof of concept using Linux ftrace capabilities on older kernels.\n\nThe output format is based on the ftrace function graph trace format,\ndescribed in the kernel source under Documentation/trace/ftrace.txt. Use the\n\\-H option to print column headings.\nNote that the output may be shuffled when different CPU buffers are read;\ncheck the CPU column for changes, or include timestamps (-t) and post sort.\n\nWARNING: This uses dynamic tracing of kernel functions, and could cause\nkernel panics or freezes. Test, and know what you are doing, before use.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE function graph, which you may already have enabled and available on\nrecent kernels. And awk.\n.SH OPTIONS\n.TP\n\\-a\nAll info. Same as \\-HPt.\n.TP\n\\-C\nFunction durations measure on-CPU time only (exclude sleep time).\n.TP\n\\-d seconds\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-H\nPrint column headers.\n.TP\n\\-p PID\nOnly trace kernel functions when this process ID is on-CPU.\n.TP\n\\-L TID\nOnly trace kernel functions when this thread ID is on-CPU.\n.TP\n\\-P\nShow process names and process IDs with every line of output.\n.TP\n\\-t\nShow timestamps on every line of output.\n.TP\nfuncstring\nA function name to trace, which may include file glob style wildcards (\"*\") at\nthe beginning or ending of a string only. Eg, \"vfs*\" means match \"vfs\" followed\nby anything. Since the output is verbose, you probably only want to trace\nsingle functions, and not use wildcards.\n.TP\nlatency_us\nMinimum function duration to trace, in units of microseconds. This is filtered\nin-kernel.\n.SH EXAMPLES\n.TP\nTrace calls to vfs_read(), showing events slower than 10 ms:\n#\n.B funcslower vfs_read 10000\n.TP\nSame as above, but include column headers, event timestamps, and process names:\n#\n.B funcslower -HPt vfs_read 10000\n.TP\nTrace slow vfs_read()s for PID 198 only:\n#\n.B funcslower \\-p 198 vfs_read 10000\n.SH FIELDS\nThe output format depends on the kernel version, and headings can be printed\nusing \\-H. The format is the same as the ftrace function trace format, described\nin the kernel source under Documentation/trace/ftrace.txt.\n\nTypical fields are:\n.TP\nTIME\n(Shown with \\-t.) Time of event, in seconds.\n.TP\nCPU\nThe CPU this event occurred on.\n.TP\nTASK/PID\n(Shown with \\-P.) The process name (which could include dashes), a dash, and the process ID.\n.TP\nDURATION\nElapsed time during the function call, inclusive of children. This is also\ninclusive of sleep time, unless -C is used.\n.TP\nFUNCTION CALLS\nKernel function returns.\n.SH OVERHEAD\nOVERHEADS: Timing and filtering is performed in-kernel context, costing\nlower overheads than post-processing in user space. If you trace frequent\nevents (eg, pick a common function and a low threshold), you might want to\ntry the \"-d secs\" option, which buffers events in-kernel instead of printing\nthem live.\n\nIt's a good idea to start with a high threshold (eg, \"100000\" for 100 ms) then\nto decrease it. If you start low instead, you may start printing too many\nevents.\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nfunccount(8), functrace(8), funcgraph(8), kprobe(8)\n"
  },
  {
    "path": "man/man8/functrace.8",
    "content": ".TH functrace 8  \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nfunctrace \\- trace kernel function calls matching specified wildcards. Uses Linux ftrace.\n.SH SYNOPSIS\n.B functrace\n[\\-hH] [\\-p PID] [\\-L TID] [\\-d secs] funcstring\n.SH DESCRIPTION\nThis tool provides a quick way to capture the execution of kernel functions,\nshowing basic details including as the process ID, timestamp, and calling\nfunction.\n\nWARNING: This uses dynamic tracing of (what can be many) kernel functions,\nand could cause kernel panics or freezes. Test, and know what you are doing,\nbefore use.\n\nAlso beware of feedback loops: tracing tcp* functions over an ssh session,\nor writing ext4* functions to an ext4 file system. For the former, tcp\ntrace data could be redirected to a file (as in the usage message). For\nthe latter, trace to the screen or a different file system.\n\nSEE ALSO: kprobe(8), which can dynamically trace a single function call or\nreturn, and examine CPU registers and return values.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE CONFIG, which you may already have enabled and available on recent\nkernels.\n.SH OPTIONS\n.TP\n\\-d seconds\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-H\nPrint column headers.\n.TP\n\\-p PID\nOnly trace kernel functions when this process ID is on-CPU.\n.TP\n\\-L TID\nOnly trace kernel functions when this thread ID is on-CPU.\n.TP\nfuncstring\nA function name to trace, which may include file glob style wildcards (\"*\") at\nthe beginning or ending of a string only. Eg, \"vfs*\" means match \"vfs\" followed\nby anything.\n.SH EXAMPLES\n.TP\nTrace calls to do_nanosleep():\n#\n.B functrace do_nanosleep\n.TP\nTrace calls to all kernel functions ending in \"*sleep\":\n#\n.B functrace '*sleep'\n.TP\nTrace all \"vfs*\" kernel function calls for PID 198:\n#\n.B functrace \\-p 198 'vfs*'\n.TP\nTrace all \"tcp*\" kernel function calls, and output to a file until Ctrl-C:\n#\n.B functrace 'tcp*' > out\n.TP\nTrace all \"tcp*\" kernel function calls, output to a file, for 1 second (buffered):\n#\n.B functrace \\-d 1 'tcp*' > out\n.SH FIELDS\nThe output format depends on the kernel version, and headings can be printed\nusing \\-H. The format is the same as the ftrace function trace format, described\nin the kernel source under Documentation/trace/ftrace.txt.\n\nTypical fields are:\n.TP\nTASK-PID\nThe process name (which could include dashes), a dash, and the process ID.\n.TP\nCPU#\nThe CPU ID, in brackets.\n.TP\n||||\nKernel state flags. For example, on Linux 3.16 these are for irqs-off,\nneed-resched, hardirq/softirq, and preempt-depth.\n.TP\nTIMESTAMP\nTime of event, in seconds.\n.TP\nFUNCTION\nKernel function name.\n.SH OVERHEAD\nThis can generate a lot of trace data quickly, depending on the\nfrequency of the traced events. Such data will cause performance overheads.\nThis also works without buffering by default, printing function events\nas they happen (uses trace_pipe), context switching and consuming CPU to do\nso. If needed, you can try the \"\\-d secs\" option, which buffers events\ninstead, reducing overhead. If you think the buffer option is losing events,\ntry increasing the buffer size (buffer_size_kb).\n\nIt's a good idea to use funccount(8) first, which is lower overhead, to\nhelp you select which functions you may want to trace using functrace(8).\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nfunccount(8), kprobe(8)\n"
  },
  {
    "path": "man/man8/iolatency.8",
    "content": ".TH iolatency 8  \"2014-07-12\" \"USER COMMANDS\"\n.SH NAME\niolatency \\- summarize block device I/O latency as a histogram. Uses Linux ftrace.\n.SH SYNOPSIS\n.B iolatency\n[\\-hQT] [\\-d device] [\\-i iotype] [interval [count]]\n.SH DESCRIPTION\nThis shows the distribution of latency, allowing modes and latency outliers\nto be identified and studied. For more details of block device I/O, use\niosnoop(8).\n\nThis is a proof of concept tool using ftrace, and involves user space\nprocessing and related overheads. See the OVERHEAD section.\n\nNOTE: Due to the way trace buffers are switched per interval, there is the\npossibility of losing a small number of I/O (usually less than 1%). The\nsummary therefore shows the general distribution, but may be slightly\nincomplete. If 100% of I/O must be studied, use iosnoop(8) and post-process.\nAlso note that I/O may be missed when the trace buffer is full: see the\ninterval section in OPTIONS.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE CONFIG, and the tracepoints block:block_rq_issue and\nblock:block_rq_complete, which you may already have enabled and available on\nrecent Linux kernels. And awk.\n.SH OPTIONS\n.TP\n\\-d device\nOnly show I/O issued by this device. (eg, \"202,1\"). This matches the DEV\ncolumn in the iolatency output, and is filtered in-kernel.\n.TP\n\\-i iotype\nOnly show I/O issued that matches this I/O type. This matches the TYPE column\nin the iolatency output, and wildcards (\"*\") can be used at the beginning or\nend (only). Eg, \"*R*\" matches all reads. This is filtered in-kernel.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-Q\nInclude block I/O queueing time. This uses block I/O queue insertion as the\nstart tracepoint (block:block_rq_insert), instead of block I/O issue\n(block:block_rq_issue).\n.TP\n\\-T\nInclude timestamps with each summary output.\n.TP\ninterval\nInterval between summary histograms, in seconds.\n\nDuring the interval, trace output will be buffered in-kernel, which is then\nread and processed for the summary. This buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size (the bufsize_kb setting in iolatency). With the\ndefault setting (4 Mbytes), I'd expect this to happen around 50k I/O per\nsummary.\n.TP\ncount\nNumber of summaries to print.\n.SH EXAMPLES\n.TP\nDefault output, print a summary of block I/O latency every 1 second:\n# \n.B iolatency\n.TP\nInclude block I/O queue time:\n.B iolatency \\-Q\n.TP\nPrint 5 x 1 second summaries:\n#\n.B iolatency 1 5\n.TP\nTrace reads only:\n#\n.B iolatency \\-i '*R*'\n.TP\nTrace I/O issued to device 202,1 only:\n#\n.B iolatency \\-d 202,1\n.SH FIELDS\n.TP\n>=(ms)\nLatency was greater than or equal-to this value, in milliseconds.\n.TP\n<(ms)\nLatency was less than this value, in milliseconds.\n.TP\nI/O\nNumber of block device I/O in this latency range, during the interval.\n.TP\nDistribution\nASCII histogram representation of the I/O column.\n.SH OVERHEAD\nBlock device I/O issue and completion events are traced and buffered\nin-kernel, then processed and summarized in user space. There may be\nmeasurable overhead with this approach, relative to the block device IOPS.\n\nThe overhead may be acceptable in many situations. If it isn't, this tool\ncan be reimplemented in C, or using a different tracer (eg, perf_events,\nSystemTap, ktap.)\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\niosnoop(8), iostat(1)\n"
  },
  {
    "path": "man/man8/iosnoop.8",
    "content": ".TH iosnoop 8  \"2014-07-12\" \"USER COMMANDS\"\n.SH NAME\niosnoop \\- trace block I/O events as they occur. Uses Linux ftrace.\n.SH SYNOPSIS\n.B iosnoop\n[\\-hQst] [\\-d device] [\\-i iotype] [\\-p pid] [\\-n name] [duration]\n.SH DESCRIPTION\niosnoop prints block device I/O events as they happen, with useful details such\nas PID, device, I/O type, block number, I/O size, and latency.\n\nThis traces disk I/O at the block device interface, using the block:\ntracepoints. This can help characterize the I/O requested for the storage\ndevices and their resulting performance. I/O completions can also be studied\nevent-by-event for debugging disk and controller I/O scheduling issues.\n\nNOTE: Use of a duration buffers I/O, which reduces overheads, but this also\nintroduces a limit to the number of I/O that will be captured. See the duration\nsection in OPTIONS.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE CONFIG, and the tracepoints block:block_rq_insert, block:block_rq_issue,\nand block:block_rq_complete, which you may already have enabled and available on\nrecent Linux kernels. And awk.\n.SH OPTIONS\n.TP\n\\-d device\nOnly show I/O issued by this device. (eg, \"202,1\"). This matches the DEV\ncolumn in the iosnoop output, and is filtered in-kernel.\n.TP\n\\-i iotype\nOnly show I/O issued that matches this I/O type. This matches the TYPE column\nin the iosnoop output, and wildcards (\"*\") can be used at the beginning or\nend (only). Eg, \"*R*\" matches all reads. This is filtered in-kernel.\n.TP\n\\-p PID\nOnly show I/O issued by this PID. This filters in-kernel. Note that I/O may be\nissued indirectly; for example, as the result of a memory allocation, causing\ndirty buffers (maybe from another PID) to be written to storage.\n\nWith the \\-Q\noption, the identified PID is more accurate, however, LATms now includes\nqueueing time (see the \\-Q option).\n.TP\n\\-n name\nOnly show I/O issued by processes with this name. Partial strings and regular\nexpressions are allowed. This is a post-filter, so all I/O is traced and then\nfiltered in user space. As with PID, this includes indirectly issued I/O,\nand \\-Q can be used to improve accuracy (see the \\-Q option).\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-Q\nUse block I/O queue insertion as the start tracepoint (block:block_rq_insert),\ninstead of block I/O issue (block:block_rq_issue). This makes the following\nchanges: COMM and PID are more likely to identify the origin process, as are\n\\-p PID and \\-n name; STARTs shows queue insert; and LATms shows I/O\ntime including time spent on the block I/O queue.\n.TP\n\\-s\nInclude a column for the start time (issue time) of the I/O, in seconds.\nIf the \\-Q option is used, this is the time the I/O is inserted on the block\nI/O queue.\n.TP\n\\-t\nInclude a column for the completion time of the I/O, in seconds.\n.TP\nduration\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size (the bufsize_kb setting in iosnoop). With the\ndefault setting (4 Mbytes), I'd expect this to happen around 50k I/O.\n.SH EXAMPLES\n.TP\nDefault output, print I/O activity as it occurs:\n# \n.B iosnoop\n.TP\nBuffer for 5 seconds (lower overhead) and write to a file:\n# \n.B iosnoop 5 > outfile\n.TP\nTrace based on block I/O queue insertion, showing queueing time:\n#\n.B iosnoop -Q\n.TP\nTrace reads only:\n#\n.B iosnoop \\-i '*R*'\n.TP\nTrace I/O issued to device 202,1 only:\n#\n.B iosnoop \\-d 202,1\n.TP\nInclude I/O start and completion timestamps:\n#\n.B iosnoop \\-ts\n.TP\nInclude I/O queueing and completion timestamps:\n#\n.B iosnop \\-Qts\n.TP\nTrace I/O issued when PID 181 was on-CPU only:\n#\n.B iosnoop \\-p 181\n.TP\nTrace I/O queued when PID 181 was on-CPU (more accurate), and include queue time:\n#\n.B iosnoop \\-Qp 181\n.SH FIELDS\n.TP\nCOMM\nProcess name (command) for the PID that was on-CPU when the I/O was issued, or\ninserted if \\-Q is used. See PID. This column is truncated to 12 characters.\n.TP\nPID\nProcess ID which was on-CPU when the I/O was issued, or inserted if \\-Q is\nused. This will usually be the\nprocess directly requesting I/O, however, it may also include indirect I/O. For\nexample, a memory allocation by this PID which causes dirty memory from another\nPID to be flushed to disk.\n.TP\nTYPE\nType of I/O. R=read, W=write, M=metadata, S=sync, A=readahead, F=flush or FUA (force unit access), D=discard, E=secure, N=null (not RWFD).\n.TP\nDEV\nStorage device ID.\n.TP\nBLOCK\nDisk block for the operation (location, relative to this device).\n.TP\nBYTES\nSize of the I/O, in bytes.\n.TP\nLATms\nLatency (time) for the I/O, in milliseconds.\n.SH OVERHEAD\nBy default, iosnoop works without buffering, printing I/O events\nas they happen (uses trace_pipe), context switching and consuming CPU to do\nso. This has a limit of about 10,000 IOPS (depending on your platform), at\nwhich point iosnoop will be consuming 1 CPU. The duration mode uses buffering,\nand can handle much higher IOPS rates, however, the buffer has a limit of\nabout 50,000 I/O, after which events will be dropped. You can tune this with\nbufsize_kb, which is per-CPU. Also note that the \"-n\" option is currently\npost-filtered, so all events are traced.\n\nThe overhead may be acceptable in many situations. If it isn't, this tool\ncan be reimplemented in C, or using a different tracer (eg, perf_events,\nSystemTap, ktap.)\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\niolatency(8), iostat(1), lsblk(8)\n"
  },
  {
    "path": "man/man8/killsnoop.8",
    "content": ".TH killsnoop 8  \"2014-09-15\" \"USER COMMANDS\"\n.SH NAME\nkillsnoop \\- trace kill() syscalls with process and signal details. Uses Linux ftrace.\n.SH SYNOPSIS\n.B killsnoop\n[\\-hst] [\\-d secs] [\\-p pid] [\\-n name]\n.SH DESCRIPTION\nThis traces kill() syscalls, showing which process killed which pid and\nreturns the returncode (0 for success, -1 for error).\n\nThis implementation is designed to work on older kernel versions, and without\nkernel debuginfo. It works by dynamic tracing of the return value of kill()\nand associating it with the previous kill() syscall return.\nThis approach is kernel version specific, and may not work on your version.\nIt is a workaround, and proof of concept for ftrace, until more kernel tracing\nfunctionality is available.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE and KPROBE CONFIG, syscalls:sys_enter_kill and\nsyscalls:sys_exit_kill kernel tracepoints (you may already have these\non recent kernels) and awk.\n.SH OPTIONS\n.TP\n\\-d secs\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-n name\nOnly show processes matching this process name. Partial strings and regular\nexpressions are allowed. This is post-filtered using awk.\n.TP\n\\-p PID\nOnly trace this process ID. This is filtered in-kernel.\n.TP\n\\-s\nUse human readable signal names, instead of signal numbers.\n.TP\n\\-t\nInclude timestamps, in seconds.\n.SH EXAMPLES\n.TP\nTrace all kill() syscalls with details:\n#\n.B killsnoop\n.TP\nTrace kill() syscalls with readable signal names, and times:\n#\n.B killsnoop -st\n.TP\nTrack kill() syscalls for processes named \"httpd\":\n#\n.B killsnoop -n httpd\n.SH FIELDS\n.TP\nTIMEs\nTime of open() completion, in units of seconds.\n.TP\nCOMM\nProcess name (if known) of the process that issued the signal.\n.TP\nPID\nProcess ID that issued the signal.\n.TP\nTPID\nTarget PID for the signal.\n.TP\nSIGNAL\nSignal number sent to the target process, or name if -s is used.\n.TP\nRETURN\nReturn status: 0 for success, -1 for failure.\n.SH OVERHEAD\nThis reads and kill() syscalls as they occur. For high rates of kills (> 500/s),\nthe overhead may begin to be measurable, however, the rate is unlikely to get\nthis high. And if it is: you should investigate why. Test yourself. You can\nalso use the \\-d mode to buffer output, reducing overheads.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nMartin Probst\n.SH SEE ALSO\ntpoint(8), execsnoop(8), opensnoop(8)\n"
  },
  {
    "path": "man/man8/kprobe.8",
    "content": ".TH kprobe 8  \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nkprobe \\- trace a given kprobe definition. Kernel dynamic tracing. Uses Linux ftrace.\n.SH SYNOPSIS\n.B kprobe\n[\\-FhHsv] [\\-d secs] [\\-p PID] [\\-L TID] kprobe_definition [filter]\n.SH DESCRIPTION\nThis will create, trace, then destroy a given kprobe definition. See\nDocumentation/trace/kprobetrace.txt in the Linux kernel source for the\nsyntax of a kprobe definition, and \"kprobe -h\" for examples. With this tool,\nthe probe alias is optional (it will become to kprobe:<funcname> if not\nspecified).\n\nWARNING: This uses dynamic tracing of kernel functions, and could cause\nkernel panics or freezes, depending on the function traced. Test in a lab\nenvironment, and know what you are doing, before use.\n\nAlso beware of feedback loops: tracing tcp functions over an ssh session,\nor writing ext4 functions to an ext4 file system. For the former, tcp\ntrace data could be redirected to a file (as in the usage message). For\nthe latter, trace to the screen or a different file system.\n\nSEE ALSO: functrace(8), which can perform basic tracing (event only) of\nmultiple kernel functions using wildcards.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE and KPROBES CONFIG, which you may already have enabled and available on\nrecent kernels.\n.SH OPTIONS\n.TP\n\\-F\nForce. Trace despite warnings. By default the specified kernel function must\nexist in the available_filter_functions file. This option overrides this check.\nThis might expose you to more unsafe functions, which could cause kernel\npanics or freezes when traced.\n.TP\n\\-d seconds\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-H\nPrint column headers.\n.TP\n\\-s\nPrint kernel stack traces after each event.\n.TP\n\\-v\nShow the kprobe format file only (do not trace), identifying possible variables\nfor use in a custom filter.\n.TP\n\\-p PID\nOnly trace kernel functions when this process ID is on-CPU.\n.TP\n\\-L TID\nOnly trace kernel functions when this thread ID is on-CPU.\n.TP\nkprobe_definition\nA full kprobe definition, as documented by Documentation/trace/kprobetrace.txt\nin the Linux kernel source. Note that the probe alias name is optional with\nkprobe(8), and if not specified, the tracepoint will become kprobe:<funcname>.\nSee the EXAMPLES section.\n.TP\nfilter\nAn ftrace filter definition.\n.SH EXAMPLES\nThese examples may need modification to match your kernel version's function\nnames and platform's register usage. If using platform specific registers\nbecomes too painful in practice, consider a kernel debuginfo-based tracer,\nwhich can trace variables names instead. For example, perf_events.\n.TP\nTrace do_sys_open() entry:\n#\n.B kprobe p:do_sys_open\n.TP\nTrace do_sys_open() return:\n#\n.B kprobe r:do_sys_open\n.TP\nTrace do_sys_open() return value:\n#\n.B kprobe 'r:do_sys_open $retval'\n.TP\nTrace do_sys_open() return value, with a custom probe alias \"myopen\":\n#\n.B kprobe 'r:myopen do_sys_open $retval'\n.TP\nTrace do_sys_open() file mode:\n#\n.B kprobe 'p:myopen do_sys_open mode=%cx:u16'\n.TP\nTrace do_sys_open() file mode for PID 81:\n#\n.B kprobe -p 81 'p:myopen do_sys_open mode=%cx:u16'\n.TP\nTrace do_sys_open() with filename string:\n#\n.B kprobe 'p:myopen do_sys_open filename=+0(%si):string'\n.TP\nTrace do_sys_open() for filenames ending in \"stat\":\n#\n.B kprobe 'p:myopen do_sys_open fn=+0(%si):string' 'fn ~ \"\"\"*stat\"\"\"'\n.TP\nTrace tcp_retransmit_skb() and show kernel stack traces, showing the path that led to it (can help explain why):\n#\n.B kprobe \\-s 'p:myprobe tcp_retransmit_skb'\n.SH FIELDS\nThe output format depends on the kernel version, and headings can be printed\nusing \\-H. The format is the same as the ftrace function trace format, described\nin the kernel source under Documentation/trace/ftrace.txt.\n\nTypical fields are:\n.TP\nTASK-PID\nThe process name (which could include dashes), a dash, and the process ID.\n.TP\nCPU#\nThe CPU ID, in brackets.\n.TP\n||||\nKernel state flags. For example, on Linux 3.16 these are for irqs-off,\nneed-resched, hardirq/softirq, and preempt-depth.\n.TP\nTIMESTAMP\nTime of event, in seconds.\n.TP\nFUNCTION\nKernel function name.\n.SH OVERHEAD\nThis can generate a lot of trace data quickly, depending on the\nfrequency of the traced events. Such data will cause performance overheads.\nThis also works without buffering by default, printing function events\nas they happen (uses trace_pipe), context switching and consuming CPU to do\nso. If needed, you can try the \"\\-d secs\" option, which buffers events\ninstead, reducing overhead. If you think the buffer option is losing events,\ntry increasing the buffer size (buffer_size_kb).\n\nIt's a good idea to use funccount(8) first, which is lower overhead, to\nhelp you select which functions you may want to trace using kprobe(8).\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nfunctrace(8), funccount(8)\n"
  },
  {
    "path": "man/man8/opensnoop.8",
    "content": ".TH opensnoop 8  \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nopensnoop \\- trace open() syscalls with file details. Uses Linux ftrace.\n.SH SYNOPSIS\n.B opensnoop\n[\\-htx] [\\-d secs] [\\-p pid] [\\-L tid] [\\-n name] [filename]\n.SH DESCRIPTION\nThis traces open() syscalls, showing the file name (pathname) and returned file\ndescriptor number (or \\-1, for error).\n\nThis implementation is designed to work on older kernel versions, and without\nkernel debuginfo. It works by dynamic tracing of the return value of getname()\nas a string, and associating it with the following open() syscall return.\nThis approach is kernel version specific, and may not work on your version.\nIt is a workaround, and proof of concept for ftrace, until more kernel tracing\nfunctionality is available.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE and KPROBE CONFIG, the syscalls:sys_exit_open tracepoint, and the\ngetname() kernel function. You may already have these enabled and available\non recent Linux kernels. And awk.\n.SH OPTIONS\n.TP\n\\-d secs\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-n name\nOnly show processes matching this process name. Partial strings and regular\nexpressions are allowed. This is post-filtered using awk.\n.TP\n\\-p PID\nOnly trace this process ID. This is filtered in-kernel.\n.TP\n\\-L TID\nOnly trace this thread ID. This is filtered in-kernel.\n.TP\n\\-t\nInclude timestamps, in seconds.\n.TP\n\\-x\nOnly print failed open()s.\n.TP\nfilename\nOnly show open()s which match this filename. Partial strings and regular\nexpressions are allowed. This is post-filtered using awk.\n.SH EXAMPLES\n.TP\nTrace all open() syscalls with details:\n#\n.B opensnoop\n.TP\nOnly trace open()s for PID 81:\n#\n.B opensnoop -p 81\n.TP\nTrace failed open() syscalls:\n#\n.B opensnoop -x\n.TP\nTrace open() syscalls for filenames containing \"conf\":\n#\n.B opensnoop conf\n.TP\nTrace open() syscalls for filenames ending in \"log\":\n#\n.B opensnoop 'log$'\n.SH FIELDS\n.TP\nTIMEs\nTime of open() completion, in units of seconds.\n.TP\nCOMM\nProcess name (if known).\n.TP\nPID\nProcess ID.\n.TP\nFD\nFile descriptor. If this is a successful open, the file descriptor number is\nshown. If this is unsuccessful, -1 is shown. Numbers beginning with 0x are\nhexadecimal.\n.TP\nFILE\nFilename (pathname) used by the open() syscall.\n.SH OVERHEAD\nThis reads and open() syscalls and getname() kernel functions as they occur.\nFor high rates of opens (> 500/s), the overhead may begin to be measurable.\nTest yourself. You can use the \\-d mode to buffer output, reducing overheads.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nexecsnoop(8), strace(1)\n"
  },
  {
    "path": "man/man8/perf-stat-hist.8",
    "content": ".TH perf-stat-hist 8  \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nperf-stat-hist \\- histogram summary of tracepoint values. Uses Linux perf_events.\n.SH SYNOPSIS\n.B perf-stat-hist\n[-h] [-b buckets|-P power] [-m max] tracepoint variable [seconds]\n.SH DESCRIPTION\nThis is a proof-of-concept showing in-kernel histograms using Linux perf_events\n(aka the \"perf\" command), on older kernels where perf_events does not have\nthis native capability.\n\nThese histograms show the distribution of variable, allowing details\nincluding multiple modes and outliers to be studied.\n\nThis uses multiple counting tracepoints with different filters, one for each\nhistogram bucket. While this is summarized in-kernel, the use of multiple\ntracepoints does add addiitonal overhead. Hopefully, in the\nfuture this this functionality will be provided in an efficient way from\nperf_events itself, at which point this tool can be deleted or rewritten.\n.SH REQUIREMENTS\nLinux perf_events: add linux-tools-common, run \"perf\", then add any additional\npackages it requests. Also uses awk.\n.SH OPTIONS\n.TP\n\\-h\nUsage message.\n.TP\n\\-b buckets\nSpecify a list of bucket points for the histogram as a string (eg, \"10 500\n1000\"). The histogram will include buckets for less-than the minimum, and\ngreater-than-or-equal-to the maximum.  If a single value is specified, two\nstatistics only are gathered: for less-than and for greater-than-or-equal-to.\nThe overhead is relative to the number of buckets, so only specifying a\nsingle value costs the lowest overhead.\n.TP\n\\-P power\nPower for power-of histogram. By default, a power-of-4 histogram is created.\nThis and the \\-b option are exclusive.\n.TP\n\\-m max\nMax value for power-of histograms.\n.TP\ntracepoint\nTracepoint specification. Eg, syscalls:sys_enter_read.\n.TP\nvariable\nThe tracepoint variable name to summarize. To see what are available, cat the\nformat file under /sys/kernel/debug/tracing/events/*/*/format.\n.TP\nseconds\nNumber of seconds to trace. If not specified, this runs until Ctrl-C.\n.SH EXAMPLES\n.TP\nTrace read() syscalls until Ctrl-C, and show histogram of requested size:\n#\n.B perf\\-stat\\-hist syscalls:sys_enter_read count\n.TP\nTrace read() syscall completions until Ctrl-C, and show histogram of successful returned size:\n#\n.B perf\\-stat\\-hist syscalls:sys_exit_read ret\n.TP\nTrace read() return sizes for 10 seconds, showing histogram:\n#\n.B perf\\-stat\\-hist syscalls:sys_exit_read ret 10\n.TP\nTrace network transmits until Ctrl-C, and show histogram of packet size:\n#\n.B perf\\-stat\\-hist net:net_dev_xmit len\n.TP\nTrace read() return sizes, using a power-of-10 histogram:\n.B perf\\-stat\\-hist \\-P 10 syscalls:sys_exit_read ret\n.TP\nTrace read() return sizes, using a power-of-2 histogram, and a max of 1024:\n.B perf\\-stat\\-hist \\-P 2 \\-m 1024 syscalls:sys_exit_read ret\n.TP\nTrace read() return sizes, using the specified bucket points:\n.B perf\\-stat\\-hist \\-b \"\"\"10 50 100 5000\"\"\" syscalls:sys_exit_read ret\n.TP\nTrace read() return sizes, and bifurcate statistics by the value 10:\n.B perf-stat-hist \\-b 10 syscalls:sys_exit_read ret\n.SH FIELDS\n.TP\nRange\nRange of the histogram bucket, in units of the variable specified.\n.TP\nCount\nNumber of occurrences (tracepoint events) of the variable in this range.\n.TP\nDistribution\nASCII histogram representation of the Count column.\n.SH OVERHEAD\nWhile the counts are performed in-kernel, there is one tracepoint used per\nhistogram bucket, so the overheads are higher than usual (relative to the\nnumber of buckets) than function counting using perf stat. The lowest\noverhead is when \\-b is used to specify one bucket only, bifurcating\nstatistics.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nperf(1)\n"
  },
  {
    "path": "man/man8/reset-ftrace.8",
    "content": ".TH reset-ftrace 8  \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nreset-ftrace \\- reset state of ftrace, disabling all tracing. Written for Linux ftrace.\n.SH SYNOPSIS\n.B reset-ftrace\n[\\-fhq]\n.SH DESCRIPTION\nThis resets the state of various ftrace files, and shows the before and after\nstate.\n\nThis may only be of use to ftrace hackers who, in the process of developing\nftrace software, often get the subsystem into a partially active state, and\nwould like a quick way to reset state. Check the end of this script for the\nactually files reset, and add more if you need.\n\nWARNING: Only use this if and when you are sure that there are no other active\nftrace sessions on your system, as otherwise it will kill them.\n.SH REQUIREMENTS\nFTRACE CONFIG.\n.SH OPTIONS\n.TP\n\\-f\nForce. If the ftrace lock file exists (/var/tmp/.ftrace-lock), delete it.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-q\nQuiet. Run, but don't print any output.\n.SH EXAMPLES\n.TP\nReset various ftrace files:\n#\n.B reset-ftrace\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nperf(1)\n"
  },
  {
    "path": "man/man8/syscount.8",
    "content": ".TH syscount 8  \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nsyscount \\- count system calls. Uses Linux perf_events.\n.SH SYNOPSIS\n.B syscount\n[\\-chv] [\\-t top] {\\-p PID|\\-d seconds|command}\n.SH DESCRIPTION\nThis is a proof-of-concept using perf_events capabilities for older kernel\nversions, that lack custom in-kernel aggregations. Once they exist, this\nscript can be substantially rewritten and improved (lower overhead).\n.SH REQUIREMENTS\nLinux perf_events: add linux-tools-common, run \"perf\", then\nadd any additional packages it requests. Also needs awk.\n.SH OPTIONS\n.TP\n\\-c\nShow counts by syscall name. This mode (without -v) uses in-kernel counts, which\nhave lower overhead than the default mode.\n.TP\n\\-h\nUsage message.\n.TP\n\\-v\nVerbose: include PID.\n.TP\n\\-p PID\nTrace this process ID only.\n.TP\n\\-d seconds\nDuration of trace in seconds.\n.TP\ncommand\nRun and trace this command.\n.SH EXAMPLES\n.TP\nTrace and summarize syscalls by process name:\n#\n.B syscount\n.TP\nTrace and summarize syscalls by syscall name (lower overhead):\n#\n.B syscount \\-c\n.TP\nTrace for 5 seconds, showing by process name:\n#\n.B syscount \\-d 5\n.TP\nTrace PID 932 only, and show by syscall name (lower overhead):\n#\n.B syscount \\-cp 923\n.TP\nExecute the \"\"\"ls\"\"\" command, and show by syscall name:\n#\n.B syscount -c ls\n.SH FIELDS\n.TP\nPID\nProcess ID.\n.TP\nCOMM\nProcess command name.\n.TP\nSYSCALL\nSyscall name.\n.TP\nCOUNT\nNumber of syscalls during tracing.\n.SH OVERHEAD\nModes that report syscall names only (\\-c, \\-cp PID, \\-cd secs) have\nlower overhead, since they use in-kernel counts. Other modes which report\nprocess IDs (\\-cv) or process names (default) create a perf.data file for\npost processing, and you will see messages about it doing this. Beware of\nthe file size (test for short durations, or use \\-c to see counts based on\nin-kernel counters), and gauge overheads based on the perf.data size.\n\nNote that this script delibrately does not pipe perf record into\nperf script, which would avoid perf.data, because it can create a feedback\nloop where the perf script syscalls are recorded. Hopefully there will be a\nfix for this in a later perf version, so perf.data can be skipped, or other\nkernel features to aggregate by process name in-kernel directly (eg, via\neBPF, ktap, or SystemTap).\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\niosnoop(8), iolatency(8), iostat(1)\n"
  },
  {
    "path": "man/man8/tcpretrans.8",
    "content": ".TH tcpretrans 8  \"2014-07-31\" \"USER COMMANDS\"\n.SH NAME\ntcpretrans \\- show TCP retransmits, with address and other details. Uses Linux ftrace.\n.SH SYNOPSIS\n.B tcpretrans\n[\\-hsp]\n.SH DESCRIPTION\nThis traces TCP retransmits that are sent by the system tcpretrans is executed\nfrom, showing address, port, and TCP state information,\nand sometimes the PID (although usually not, since retransmits are usually\nsent by the kernel on timeout events). To keep overhead low, only\ntcp_retransmit_skb() kernel calls are traced (this does not trace every packet).\n\nThis was written as a proof of concept for ftrace, for older Linux systems,\nand without kernel debuginfo. It uses dynamic tracing of tcp_retransmit_skb(),\nand reads /proc/net/tcp for socket details. Its use of dynamic tracing and\nCPU registers is an unstable platform-specific workaround, and may require\nmodifications to work on different kernels and platforms. This would be better\nwritten using a tracer such as SystemTap, and will likely be rewritten in the\nfuture when certain tracing features are added to the Linux kernel.\n\nWhen \\-l is used, this also uses dynamic tracing of tcp_send_loss_probe() and\na register.\n\nCurrently only IPv4 is supported, on x86_64. If you try this on a different\narchitecture, you'll likely need to adjust the register locations (search\nfor %di).\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE and KPROBE CONFIG, tcp_retransmit_skb() kernel function.\nYou may have these already have these on recent kernels. And Perl.\nTCP tail loss probes were added in Linux 3.10.\n.SH OPTIONS\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-s\nInclude kernel stack traces.\n.TP\n\\-l\nInclude TCP tail loss probes.\n.SH EXAMPLES\n.TP\nTrace TCP retransmits\n#\n.B tcpretrans\n.TP\nTIME\nTime of retransmit (may be rounded up to the nearest second).\n.TP\nPID\nProcess ID that was on-CPU. This is less useful than it might sound, as it\nmay usually be 0, for the kernel, for timer-based retransmits.\n.TP\nLADDR\nLocal address.\n.TP\nLPORT\nLocal port.\n.TP\n\\-\\-\nPacket type: \"R>\" for retransmit, and \"L>\" for tail loss probe.\n.TP\nRADDR\nRemote address.\n.TP\nRPORT\nRemote port.\n.TP\nSTATE\nTCP session state.\n.SH OVERHEAD\nThe CPU overhead is relative to the rate of TCP retransmits, and is\ndesigned to be low as this does not examine every packet. Once per second the\n/proc/net/tcp file is read, and a buffer of retransmit trace events is\nretrieved from the kernel and processed.\n.SH SOURCE\nThis is from the perf-tools collection.\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\ntcpdump(1)\n"
  },
  {
    "path": "man/man8/tpoint.8",
    "content": ".TH tpoint 8  \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\ntpoint \\- trace a given tracepoint. Static tracing. Uses Linux ftrace.\n.SH SYNOPSIS\n.B tpoint\n[\\-hHsv] [\\-d secs] [\\-p PID] [\\-L TID] tracepoint [filter]\n\n.B tpoint\n\\-l\n.SH DESCRIPTION\nThis will enable a given tracepoint, print events, then disable the tracepoint\nwhen the program ends. This is like a simple version of the \"perf\" command for\nprinting live tracepoint events only. Wildcards are currently not supported.\nIf for any reason tpoint(8) is insufficient, use the more powerful perf\ncommand for tracing tracepoints instead.\n\nBeware of feedback loops: tracing tcp functions over an ssh session,\nor writing ext4 events to an ext4 file system. For the former, tcp\ntrace data could be redirected to a file (as in the usage message). For\nthe latter, trace to the screen or a different file system.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nFTRACE CONFIG and tracepoints, which you may already have enabled and available\non recent kernels.\n.SH OPTIONS\n.TP\n\\-d seconds\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-H\nPrint column headers.\n.TP\n\\-l\nList tracepoints only.\n.TP\n\\-s\nPrint kernel stack traces after each event.\n.TP\n\\-v\nShow the tpoint format file only (do not trace), identifying possible variables\nfor use in a custom filter.\n.TP\n\\-p PID\nOnly trace kernel functions when this process ID is on-CPU.\n.TP\n\\-L TID\nOnly trace kernel functions when this thread ID is on-CPU.\n.TP\ntracepoint\nA tracepoint name. Eg, block:block_rq_issue. See the EXAMPLES section.\n.TP\nfilter\nAn ftrace filter definition.\n.SH EXAMPLES\n.TP\nList tracepoints containing \"open\":\n#\n.B tpoint -l | grep open\n.TP\nTrace open() syscall entry:\n#\n.B tpoint syscalls:sys_enter_open\n.TP\nTrace open() syscall entry, showing column headers:\n#\n.B tpoint -H syscalls:sys_enter_open\n.TP\nTrace block I/O issue:\n#\n.B tpoint block:block_rq_issue\n.TP\nTrace block I/O issue with stack traces:\n#\n.B tpoint \\-s block:block_rq_issue\n.SH FIELDS\nThe output format depends on the kernel version, and headings can be printed\nusing \\-H. The format is the same as the ftrace function trace format, described\nin the kernel source under Documentation/trace/ftrace.txt.\n\nTypical fields are:\n.TP\nTASK-PID\nThe process name (which could include dashes), a dash, and the process ID.\n.TP\nCPU#\nThe CPU ID, in brackets.\n.TP\n||||\nKernel state flags. For example, on Linux 3.16 these are for irqs-off,\nneed-resched, hardirq/softirq, and preempt-depth.\n.TP\nTIMESTAMP\nTime of event, in seconds.\n.TP\nFUNCTION\nKernel function name.\n.SH OVERHEAD\nThis can generate a lot of trace data quickly, depending on the\nfrequency of the traced events. Such data will cause performance overheads.\nThis also works without buffering by default, printing function events\nas they happen (uses trace_pipe), context switching and consuming CPU to do\nso. If needed, you can try the \"\\-d secs\" option, which buffers events\ninstead, reducing overhead. If you think the buffer option is losing events,\ntry increasing the buffer size (buffer_size_kb).\n\nBefore using tpoint(8), you can use perf_events to count the rate of events\nfor the tracepoint of interest, to gauge overhead. For example:\n\n.B perf stat \\-e block:block_rq_issue \\-a sleep 5\n\nThat counts the occurrences of the block:block_rq_issue tracepoint for\n5 seconds.\n\nAlso consider using perf_events, which manages buffers differently and more\nefficiently, for higher frequency applications.\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nfunctrace(8), funccount(8), perf(1)\n"
  },
  {
    "path": "man/man8/uprobe.8",
    "content": ".TH uprobe 8  \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nuprobe \\- trace a given uprobe definition. User-level dynamic tracing. Uses Linux ftrace. EXPERIMENTAL.\n.SH SYNOPSIS\n.B uprobe\n[\\-FhHsv] [\\-d secs] [\\-p PID] [\\-L TID] {\\-l target | uprobe_definition [filter]}\n.SH DESCRIPTION\nThis will create, trace, then destroy a given uprobe definition. See\nDocumentation/trace/uprobetracer.txt in the Linux kernel source for the\nsyntax of a uprobe definition, and \"uprobe -h\" for examples. With this tool,\nthe probe alias is optional (it will default to something meaningful).\n\nWARNING: This uses dynamic tracing of user-level functions, using some\nrelatively new kernel code. I have seen this cause target processes to fail,\neither entering endless spin loops or crashing on illegal instructions. I\nbelieve newer kernels (post 4.0) are relatively safer, but use caution. Test\nin a lab environment, and know what you are doing, before use. Also consider\nother (more developed) user-level tracers (perf_events, LTTng, etc.).\n\nUse extreme caution with the raw address mode: eg, \"p:libc:0xbf130\". uprobe\ndoes not check for instruction alignment, so tracing the wrong address (eg,\nmid-way through a multi-byte instruction) will corrupt the target's memory.\nOther tracers (eg, perf_events with debuginfo) check alignment.\n\nAlso beware of widespread tracing that interferes with the operation of the\nsystem, eg, tracing libc:malloc, which by-default will trace _all_ processes.\n\nI wrote this because I kept testing different custom uprobes at the command\nline, and wanted a way to automate the steps. For generic user-level\ntracing, use perf_events directly.\n\nSince this uses ftrace, only the root user can use this tool.\n.SH REQUIREMENTS\nREQUIREMENTS: FTRACE and UPROBE CONFIG, which you may already have on recent\nkernel versions, file(1), ldconfig(8), objdump(1), and some version of awk.\nAlso, currently only executes on Linux 4.0+ (see WARNING) unless -F is used.\n.SH OPTIONS\n.TP\n\\-F\nForce. Trace despite kernel version warnings. Use on older kernels may expose\nyou to (since fixed) bugs, which can lock up or crash target processes, which\ncould also lock up the entire system. Test in a lab environment before use,\nand consider other more developed user-level tracers (perf_events, LTTng,\netc.).\n.TP\n\\-d seconds\nSet the duration of tracing, in seconds. Trace output will be buffered and\nprinted at the end. This also reduces overheads by buffering in-kernel,\ninstead of printing events as they occur.\n\nThe ftrace buffer has a fixed size per-CPU (see\n/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,\ntry increasing that size.\n.TP\n\\-h\nPrint usage message.\n.TP\n\\-H\nPrint column headers.\n.TP\n\\-s\nPrint user-level stack traces after each event. These are currently printed\nin hex, and need post-processing to see user-level symbols (eg, addr2line;\nI should automate that).\n.TP\n\\-v\nShow the uprobe format file only (do not trace), identifying possible variables\nfor use in a custom filter.\n.TP\n\\-p PID\nOnly trace user-level functions when this process ID is on-CPU.\n.TP\n\\-L TID\nOnly trace user-level functions when this thread ID is on-CPU.\n.TP\nuprobe_definition\nA full uprobe definition, as documented by Documentation/trace/uprobetracer.txt\nin the Linux kernel source. Note that the probe alias name is optional with\nuprobe(8), and if not specified, it will default to something meaningful.\nSee the EXAMPLES section.\n.TP\nfilter\nAn ftrace filter definition.\n.SH EXAMPLES\nThese examples may need modification to match your target software function\nnames and platform's register usage. If using platform specific registers\nbecomes too painful in practice, consider a debuginfo-based tracer,\nwhich can trace variables names instead (eg, perf_events).\n.TP\ntrace readline() calls in all running \"bash\" executables:\n#\n.B uprobe p:bash:readline\n.TP\ntrace readline() with explicit executable path:\n#\n.B uprobe p:/bin/bash:readline\n.TP\ntrace the return of readline() with return value as a string:\n#\n.B uprobe 'r:bash:readline +0($retval):string'\n.TP\ntrace sleep() calls in all running libc shared libraries:\n#\n.B uprobe p:libc:sleep\n.TP\ntrace sleep() with register %di (x86):\n#\n.B uprobe 'p:libc:sleep %di'\n.TP\ntrace this address (use caution: must be instruction aligned):\n#\n.B uprobe p:libc:0xbf130\n.TP\ntrace gettimeofday() for PID 1182 only:\n#\n.B uprobe -p 1182 p:libc:gettimeofday\n.TP\ntrace the return of fopen() only when it returns NULL:\n#\n.B uprobe 'r:libc:fopen file=$retval' 'file == 0'\n.SH FIELDS\nThe output format depends on the kernel version, and headings can be printed\nusing \\-H. The format is the same as the ftrace function trace format, described\nin the kernel source under Documentation/trace/ftrace.txt.\n\nTypical fields are:\n.TP\nTASK-PID\nThe process name (which could include dashes), a dash, and the process ID.\n.TP\nCPU#\nThe CPU ID, in brackets.\n.TP\n||||\nKernel state flags. For example, on Linux 3.16 these are for irqs-off,\nneed-resched, hardirq/softirq, and preempt-depth.\n.TP\nTIMESTAMP\nTime of event, in seconds.\n.TP\nFUNCTION\nUser-level function name.\n.SH OVERHEAD\nThis can generate a lot of trace data quickly, depending on the\nfrequency of the traced events. Such data will cause performance overheads.\nThis also works without buffering by default, printing function events\nas they happen (uses trace_pipe), context switching and consuming CPU to do\nso. If needed, you can try the \"\\-d secs\" option, which buffers events\ninstead, reducing overhead. If you think the buffer option is losing events,\ntry increasing the buffer size (buffer_size_kb).\n\nIf you find a use for uprobe(8) where the overhead is prohibitive, consider\nthe same enabling using perf_events where overhead should be reduced.\n.SH SOURCE\nThis is from the perf-tools collection:\n.IP\nhttps://github.com/brendangregg/perf-tools\n.PP\nAlso look under the examples directory for a text file containing example\nusage, output, and commentary for this tool.\n.SH OS\nLinux\n.SH STABILITY\nUnstable - in development.\n.SH AUTHOR\nBrendan Gregg\n.SH SEE ALSO\nkprobe(8)\n"
  },
  {
    "path": "misc/perf-stat-hist",
    "content": "#!/bin/bash\n#\n# perf-stat-hist - perf_events stat histogram hack.\n#                  Written using Linux perf_events (aka \"perf\").\n#\n# This is a proof-of-concept showing in-kernel histogram summaries of a\n# tracepoint variable.\n#\n# USAGE: perf-stat-hist [-h] [-b buckets|-P power] [-m max] tracepoint\n#        variable [seconds]\n#\n# Run \"perf-stat-hist -h\" for full usage.\n#\n# This uses multiple counting tracepoints with different filters, one for each\n# histogram bucket. While this is summarized in-kernel, the use of multiple\n# tracepoints does add addiitonal overhead, which is more evident if you change\n# the power-of size from 4 to 2 (which creates more buckets). Hopefully, in the\n# future this this functionality will be provided in an efficient way from\n# perf_events itself, at which point this tool can be rewritten.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n# \n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 30-Jun-2014\tBrendan Gregg\tCreated this.\n\nopt_buckets=0; buckets=; opt_power=0; power=4; opt_max=0; max=$((1024 * 1024))\nopt_filter=0; filter=; duration=0; debug=0\ntrap ':' INT QUIT TERM PIPE HUP\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: perf-stat-hist [-h] [-b buckets|-P power] [-m max] [-f filter]\n\t                      tracepoint variable [seconds]\n\t                 -b buckets      # specify histogram bucket points\n\t                 -P power        # power-of (default is 4)\n\t                 -m max          # max value for power-of\n\t                 -f filter       # specify a filter \n\t                 -h              # this usage message\n\t   eg,\n\t       perf-stat-hist syscalls:sys_enter_read count 5\n\t                 # read() request histogram, 5 seconds\n\t       perf-stat-hist syscalls:sys_exit_read ret 5\n\t                 # read() return histogram, 5 seconds\n\t       perf-stat-hist -P 10 syscalls:sys_exit_read ret 5\n\t                 # ... use power-of-10\n\t       perf-stat-hist -P 2 -m 1024 syscalls:sys_exit_read ret 5\n\t                 # ... use power-of-2, max 1024\n\t       perf-stat-hist -b \"10 50 100 500\" syscalls:sys_exit_read ret 5\n\t                 # ... histogram based on these bucket ranges\n\t       perf-stat-hist -b 10 syscalls:sys_exit_read ret 5\n\t                 # ... bifurcate by the value 10 (lowest overhead)\n\t       perf-stat-hist -f 'rwbs == \"WS\"' block:block_rq_complete nr_sector 5\n\t                 # ... synchronous writes histogram, 5 seconds\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\n### process options\nwhile getopts b:hm:P:f: opt\ndo\n\tcase $opt in\n\tb)\topt_buckets=1; buckets=($OPTARG) ;;\n\tP)\topt_power=1; power=$OPTARG ;;\n\tm)\topt_max=1; max=$OPTARG ;;\n\tf)\topt_filter=1; filter=\"$OPTARG && \" ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n(( $# < 2 )) && usage\ntpoint=$1\t\t\t# tracepoint\nvar=$2\t\t\t\t# variable for histogram\nduration=${3}\n\n### option logic\n(( opt_buckets && opt_power )) && die \"ERROR: use either -b or -P\"\n(( opt_power && power < 2 )) && die \"ERROR: -P power must be 2 or higher\"\n\n### check that tracepoint exists\nif ! grep \"^$tpoint\\$\" /sys/kernel/debug/tracing/available_events > /dev/null\nthen\n\techo >&2 \"ERROR: tracepoint \\\"$tpoint\\\" not found. Exiting...\"\n\t[[ \"$USER\" != \"root\" ]] && echo >&2 \"Not root user?\"\n\texit 1\nfi\n\n### auto build power-of buckets\nif (( !opt_buckets )); then\n\tb=0\n\ts=1\n\twhile (( s <= max )); do\n\t\tb=\"$b $s\"\n\t\t(( s *= power ))\n\tdone\n\tbuckets=($b)\nfi\n\n### build list of tracepoints and filters for each histogram bucket\nmax=${buckets[${#buckets[@]} - 1]}\t# last element\n((max_i = ${#buckets[*]} - 1))\ntpoints=\"-e $tpoint --filter \\\"$filter $var < ${buckets[0]}\\\"\"\nawkarray=\ni=0\nwhile (( i < max_i )); do\n\tif (( i && ${buckets[$i]} <= ${buckets[$i - 1]} )); then\n\t\tdie \"ERROR: bucket list must increase in size.\"\n\tfi\n\ttpoints=\"$tpoints -e $tpoint --filter \\\"$filter $var >= ${buckets[$i]} && \"\n\ttpoints=\"$tpoints $var < ${buckets[$i + 1]}\\\"\"\n\tawkarray=\"$awkarray buckets[$i]=${buckets[$i]};\"\n\t(( i++ ))\ndone\nawkarray=\"$awkarray buckets[$max_i]=${buckets[$max_i]};\"\ntpoints=\"$tpoints -e $tpoint --filter \\\"$filter $var >= ${buckets[$max_i]}\\\"\"\n\nif (( debug )); then\n\techo buckets: ${buckets[*]}\n\techo tracepoints: $tpoints\n\techo awkarray: ${awkarray[*]}\nfi\n\n### prepare to run\nif (( duration )); then\n\tetext=\"for $duration seconds\"\n\tcmd=\"sleep $duration\"\nelse\n\tetext=\"until Ctrl-C\"\n\tcmd=\"sleep 999999\"\nfi\n\np_tpoint=$tpoint\nif [ -n \"$filter\" ]; then \n\tp_tpoint=\"$tpoint (Filter: ${filter%????})\"\nfi\n\nif (( opt_buckets )); then\n\techo \"Tracing $p_tpoint, specified buckets, $etext...\"\nelse\n\techo \"Tracing $p_tpoint, power-of-$power, max $max, $etext...\"\nfi\n\n### run perf\nout=\"-o /dev/stdout\"\t# a workaround needed in linux 3.2; not by 3.4.15\nstat=$(eval perf stat $tpoints -a $out $cmd 2>&1)\nif (( $? != 0 )); then\n\techo >&2 \"ERROR running perf:\"\n\techo >&2 \"$stat\"\n\texit\nfi\n\nif (( debug )); then\n\techo raw output:\n\techo \"$stat\"\n\techo\nfi\n\n### find max value for ASCII histogram\nmost=$(echo \"$stat\" | awk -v tpoint=$tpoint '\n\t$2 == tpoint { gsub(/,/, \"\"); if ($1 > m) { m = $1 } }\n\tEND { print m }'\n)\n\n### process output\necho\necho \"$stat\" | awk -v tpoint=$tpoint -v max_i=$max_i -v most=$most '\n\tfunction star(sval, smax, swidth) {\n\t\tstars = \"\"\n\t\tif (smax == 0) return \"\"\n\t\tfor (si = 0; si < (swidth * sval / smax); si++) {\n\t\t\tstars = stars \"#\"\n\t\t}\n\t\treturn stars\n\t}\n\tBEGIN {\n\t\t'\"$awkarray\"'\n\t\tprintf(\"            %-15s: %-8s %s\\n\", \"Range\", \"Count\",\n\t\t    \"Distribution\")\n\t}\n\t/Performance counter stats/ { i = -1 }\n\t# reverse order of rule set is important\n\t{ ok = 0 }\n\t$2 == tpoint { num = $1; gsub(/,/, \"\", num); ok = 1 }\n\tok && i >= max_i {\n\t\tprintf(\"   %10d -> %-10s: %-8s |%-38s|\\n\", buckets[i],\n\t\t    \"\", num, star(num, most, 38))\n\t\tnext\n\t}\n\tok && i >= 0 && i < max_i {\n\t\tprintf(\"   %10d -> %-10d: %-8s |%-38s|\\n\", buckets[i],\n\t\t    buckets[i+1] - 1, num, star(num, most, 38))\n\t\ti++\n\t\tnext\n\t}\n\tok && i == -1 {\n\t\tprintf(\"   %10s -> %-10d: %-8s |%-38s|\\n\", \"\",\n\t\t    buckets[0] - 1, num, star(num, most, 38))\n\t\ti++\n\t}\n'\n"
  },
  {
    "path": "net/tcpretrans",
    "content": "#!/usr/bin/perl\n#\n# tcpretrans - show TCP retransmts, with address and other details.\n#              Written using Linux ftrace.\n#\n# This traces TCP retransmits, showing address, port, and TCP state information,\n# and sometimes the PID (although usually not, since retransmits are usually\n# sent by the kernel on timeouts). To keep overhead low, only\n# tcp_retransmit_skb() calls are traced (this does not trace every packet).\n#\n# USAGE: ./tcpretrans [-hls]\n#\n# REQUIREMENTS: FTRACE and KPROBE CONFIG, tcp_retransmit_skb() kernel function,\n# and tcp_send_loss_probe() when -l is used. You may have these already have\n# these on recent kernels. And Perl.\n#\n# This was written as a proof of concept for ftrace, for older Linux systems,\n# and without kernel debuginfo. It uses dynamic tracing of tcp_retransmit_skb(),\n# and reads /proc/net/tcp for socket details. Its use of dynamic tracing and\n# CPU registers is an unstable platform-specific workaround, and may require\n# modifications to work on different kernels and platforms. This would be better\n# written using a tracer such as SystemTap, and will likely be rewritten in the\n# future when certain tracing features are added to the Linux kernel.\n#\n# When -l is used, this also uses dynamic tracing of tcp_send_loss_probe() and\n# a register.\n#\n# Currently only IPv4 is supported, on x86_64. If you try this on a different\n# architecture, you'll likely need to adjust the register locations (search\n# for %di).\n#\n# OVERHEAD: The CPU overhead is relative to the rate of TCP retransmits, and is\n# designed to be low as this does not examine every packet. Once per second the\n# /proc/net/tcp file is read, and a buffer of retransmit trace events is\n# retrieved from the kernel and processed.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the tcpretrans(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 28-Jul-2014\tBrendan Gregg\tCreated this.\n\nuse strict;\nuse warnings;\nuse POSIX qw(strftime);\nuse Getopt::Long;\nmy $tracing = \"/sys/kernel/debug/tracing\";\nmy $flock = \"/var/tmp/.ftrace-lock\";\nmy $interval = 1;\nlocal $SIG{INT} = \\&cleanup;\nlocal $SIG{QUIT} = \\&cleanup;\nlocal $SIG{TERM} = \\&cleanup;\nlocal $SIG{PIPE} = \\&cleanup;\nlocal $SIG{HUP} = \\&cleanup;\n$| = 1;\n\n### options\nmy ($help, $stacks, $tlp);\nGetOptions(\"help|h\"   => \\$help,\n\t   \"stacks|s\" => \\$stacks,\n\t   \"tlp|l\" => \\$tlp)\nor usage();\nusage() if $help;\n\nsub usage {\n\tprint STDERR \"USAGE: tcpretrans [-hls]\\n\";\n\tprint STDERR \"                  -h      # help message\\n\";\n\tprint STDERR \"                  -l      # trace TCP tail loss probes\\n\";\n\tprint STDERR \"                  -s      # print stack traces\\n\";\n\tprint STDERR \"   eg,\\n\";\n\tprint STDERR \"       tcpretrans         # trace TCP retransmits\\n\";\n\texit;\n}\n\n# delete lock and die\nsub ldie {\n\tunlink $flock;\n\tdie @_;\n}\n\n# end tracing (silently) and die\nsub edie {\n\tprint STDERR \"@_\\n\";\n\tclose STDOUT;\n\tclose STDERR;\n\tcleanup();\n}\n\nsub writeto {\n\tmy ($string, $file) = @_;\n\topen FILE, \">$file\" or return 0;\n\tprint FILE $string or return 0;\n\tclose FILE or return 0;\n}\n\nsub appendto {\n\tmy ($string, $file) = @_;\n\topen FILE, \">>$file\" or return 0;\n\tprint FILE $string or return 0;\n\tclose FILE or return 0;\n}\n\n# kprobe functions\nsub create_kprobe {\n\tmy ($kname, $kval) = @_;\n\tappendto \"p:$kname $kval\", \"kprobe_events\" or return 0;\n}\n\nsub enable_kprobe {\n\tmy ($kname) = @_;\n\twriteto \"1\", \"events/kprobes/$kname/enable\" or return 0;\n}\n\nsub remove_kprobe {\n\tmy ($kname) = @_;\n\twriteto \"0\", \"events/kprobes/$kname/enable\" or return 0;\n\tappendto \"-:$kname\", \"kprobe_events\" or return 0;\n}\n\n# tcp socket cache\nmy %tcp;\nsub cache_tcp {\n\tundef %tcp;\n\topen(TCP, \"/proc/net/tcp\") or ldie \"ERROR: reading /proc/net/tcp.\";\n\twhile (<TCP>) {\n\t\tnext if /^ *sl/;\n\t\tmy ($sl, $local_address, $rem_address, $st, $tx_rx, $tr_tm,\n\t\t    $retrnsmt, $uid, $timeout, $inode, $jf, $sk) = split;\n\t\t$sk =~ s/^0x//;\n\t\t$tcp{$sk}{laddr} = $local_address;\n\t\t$tcp{$sk}{raddr} = $rem_address;\n\t\t$tcp{$sk}{state} = $st;\n\t}\n\tclose TCP;\n}\n\nmy @tcpstate;\nsub map_tcp_states {\n\tpush @tcpstate, \"NULL\";\n\tfor (<DATA>) {\n\t\tchomp;\n\t\ts/.*TCP_//;\n\t\ts/[, ].*$//;\n\t\tpush @tcpstate, $_;\n\t}\n}\n\n# /proc/net/tcp hex addr to dotted quad decimal\nsub inet_h2a {\n\tmy ($haddr) = @_;\n\n\tmy @addr = ();\n\tfor my $num ($haddr =~ /(..)(..)(..)(..)/) {\n\t\tunshift @addr, hex($num);\n\t}\n\treturn join(\".\", @addr);\n}\n\n### check permissions\nchdir \"$tracing\" or die \"ERROR: accessing tracing. Root? Kernel has FTRACE?\" .\n    \"\\ndebugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\";\n\n### ftrace lock\nif (-e $flock) {\n\topen FLOCK, $flock; my $fpid = <FLOCK>; chomp $fpid; close FLOCK;\n\tdie \"ERROR: ftrace may be in use by PID $fpid ($flock)\";\n}\nwriteto \"$$\", $flock or die \"ERROR: unable to write $flock.\";\n\n#\n# Setup and begin tracing.\n# Use of ldie() and edie() ensures that if an error is encountered, the\n# kernel is not left in a partially configured state.\n#\nwriteto \"nop\", \"current_tracer\" or ldie \"ERROR: disabling current_tracer.\";\nmy $kname_rtr = \"tcpretrans_tcp_retransmit_skb\";\nmy $kname_tlp = \"tcpretrans_tcp_send_loss_probe\";\ncreate_kprobe $kname_rtr, \"tcp_retransmit_skb sk=%di\" or\n    ldie \"ERROR: creating kprobe for tcp_retransmit_skb().\";;\nif ($tlp) {\n\tcreate_kprobe $kname_tlp, \"tcp_send_loss_probe sk=%di\" or\n\t    edie \"ERROR: creating kprobe for tcp_send_loss_probe(). \" .\n\t         \"Older kernel version?\";\n}\nif ($stacks) {\n\twriteto \"1\", \"options/stacktrace\" or print STDERR \"WARNING: \" .\n\t    \"unable to enable stacktraces.\\n\";\n}\nenable_kprobe $kname_rtr or edie \"ERROR: enabling $kname_rtr probe.\";\nif ($tlp) {\n\tenable_kprobe $kname_tlp or edie \"ERROR: enabling $kname_tlp probe.\";\n}\nmap_tcp_states();\nprintf \"%-8s %-6s %-20s -- %-20s %-12s\\n\", \"TIME\", \"PID\", \"LADDR:LPORT\",\n    \"RADDR:RPORT\", \"STATE\";\n\n#\n# Read and print event data. This loop waits one second then reads the buffered\n# trace data, then caches /proc/net/tcp, then iterates over the buffered trace\n# data using the cached state. While this minimizes CPU overheads, it only\n# works because sockets that are retransmitting are usually long lived, and\n# remain in /proc/net/tcp for at least our sleep interval.\n#\nwhile (1) {\n\tsleep $interval;\n\n\t# buffer trace data\n\topen TPIPE, \"trace\" or edie \"ERROR: opening trace_pipe.\";\n\tmy @trace = ();\n\twhile (<TPIPE>) {\n\t\tnext if /^#/;\n\t\tpush @trace, $_;\n\t}\n\tclose TPIPE;\n\twriteto \"0\", \"trace\" or edie \"ERROR: clearing trace\";\n\n\t# cache /proc/net/tcp state\n\tif (scalar @trace) {\n\t\tcache_tcp();\n\t}\n\n\t# process and print events\n\tfor (@trace) {\n\t\tif ($stacks && /^ *=>/) {\n\t\t\tprint $_;\n\t\t\tnext;\n\t\t}\n\n\t\tmy ($taskpid, $rest) = split ' ', $_, 2;\n\t\tmy ($task, $pid) = $taskpid =~ /(.*)-(\\d+)/;\n\n\t\tmy ($skp) = $rest =~ /sk=([0-9a-fx]*)/;\n\t\tnext unless defined $skp and $skp ne \"\";\n\t\t$skp =~ s/^0x//;\n\n\t\tmy ($laddr, $lport, $raddr, $rport, $state);\n\t\tif (defined $tcp{$skp}) {\n\t\t\t# convert /proc/net/tcp hex to dotted quads\n\t\t\tmy ($hladdr, $hlport) = split /:/, $tcp{$skp}{laddr};\n\t\t\tmy ($hraddr, $hrport) = split /:/, $tcp{$skp}{raddr};\n\t\t\t$laddr = inet_h2a($hladdr);\n\t\t\t$raddr = inet_h2a($hraddr);\n\t\t\t$lport = hex($hlport);\n\t\t\t$rport = hex($hrport);\n\t\t\t$state = $tcpstate[hex($tcp{$skp}{state})];\n\t\t} else {\n\t\t\t# socket closed too quickly\n\t\t\t($laddr, $raddr) = (\"-\", \"-\");\n\t\t\t($lport, $rport) = (\"-\", \"-\");\n\t\t\t$state = \"-\";\n\t\t}\n\n\t\tmy $now = strftime \"%H:%M:%S\", localtime;\n\t\tprintf \"%-8s %-6s %-20s %s> %-20s %-12s\\n\", $now, $pid,\n\t\t    \"$laddr:$lport\", $rest =~ /$kname_tlp/ ? \"L\" : \"R\",\n\t\t    \"$raddr:$rport\", $state,\n\t}\n}\n\n### end tracing\ncleanup();\n\nsub cleanup {\n\tprint \"\\nEnding tracing...\\n\";\n\tclose TPIPE;\n\tif ($stacks) {\n\t\twriteto \"0\", \"options/stacktrace\" or print STDERR \"WARNING: \" .\n\t\t    \"unable to disable stacktraces.\\n\";\n\t}\n\tremove_kprobe $kname_rtr\n\t    or print STDERR \"ERROR: removing kprobe $kname_rtr\\n\";\n\tif ($tlp) {\n\t\tremove_kprobe $kname_tlp\n\t\t    or print STDERR \"ERROR: removing kprobe $kname_tlp\\n\";\n\t}\n\twriteto \"\", \"trace\";\n\tunlink $flock;\n\texit;\n}\n\n# from /usr/include/netinet/tcp.h:\n__DATA__\n  TCP_ESTABLISHED = 1,\n  TCP_SYN_SENT,\n  TCP_SYN_RECV,\n  TCP_FIN_WAIT1,\n  TCP_FIN_WAIT2,\n  TCP_TIME_WAIT,\n  TCP_CLOSE,\n  TCP_CLOSE_WAIT,\n  TCP_LAST_ACK,\n  TCP_LISTEN,\n  TCP_CLOSING   /* now a valid state */\n"
  },
  {
    "path": "opensnoop",
    "content": "#!/bin/bash\n#\n# opensnoop - trace open() syscalls with file details.\n#             Written using Linux ftrace.\n#\n# This traces open() syscalls, showing the file name and returned file\n# descriptor number (or -1, for error).\n#\n# This implementation is designed to work on older kernel versions, and without\n# kernel debuginfo. It works by dynamic tracing of the return value of getname()\n# as a string, and associating it with the following open() syscall return.\n# This approach is kernel version specific, and may not work on your version.\n# It is a workaround, and proof of concept for ftrace, until more kernel tracing\n# functionality is available.\n#\n# USAGE: ./opensnoop [-htx] [-d secs] [-p pid] [-L tid] [-n name] [filename]\n#\n# Run \"opensnoop -h\" for full usage.\n#\n# REQUIREMENTS: FTRACE and KPROBE CONFIG, syscalls:sys_exit_open tracepoint,\n# getname() kernel function (you may already have these on recent kernels),\n# and awk.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the opensnoop(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 20-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock; wroteflock=0\nopt_duration=0; duration=; opt_name=0; name=; opt_pid=0; pid=; opt_tid=0; tid=\nftext=; opt_time=0; opt_fail=0; opt_file=0; file=\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: opensnoop [-htx] [-d secs] [-p PID] [-L TID] [-n name] [filename]\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -n name         # process name to match on open\n\t                 -p PID          # PID to match on open\n\t                 -L TID          # PID to match on open\n\t                 -t              # include time (seconds)\n\t                 -x              # only show failed opens\n\t                 -h              # this usage message\n\t                 filename        # match filename (partials, REs, ok)\n\t  eg,\n\t       opensnoop                 # watch open()s live (unbuffered)\n\t       opensnoop -d 1            # trace 1 sec (buffered)\n\t       opensnoop -p 181          # trace I/O issued by PID 181 only\n\t       opensnoop conf            # trace filenames containing \"conf\"\n\t       opensnoop 'log$'          # filenames ending in \"log\"\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > events/kprobes/getnameprobe/enable\"\n\twarn \"echo 0 > events/syscalls/sys_exit_open/enable\"\n\twarn \"echo 0 > events/syscalls/sys_exit_openat/enable\"\n\tif (( opt_pid || opt_tid )); then\n\t\twarn \"echo 0 > events/kprobes/getnameprobe/filter\"\n\t\twarn \"echo 0 > events/syscalls/sys_exit_open/filter\"\n\t\twarn \"echo 0 > events/syscalls/sys_exit_openat/filter\"\n\tfi\n\twarn \"echo -:getnameprobe >> kprobe_events\"\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts d:hn:p:L:tx opt\ndo\n\tcase $opt in\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tn)\topt_name=1; name=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tL)\topt_tid=1; tid=$OPTARG ;;\n\tt)\topt_time=1 ;;\n\tx)\topt_fail=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\nif (( $# )); then\n\topt_file=1\n\tfile=$1\n\tshift\nfi\n(( $# )) && usage\n\n### option logic\n(( opt_pid + opt_name + opt_tid > 1 )) && \\\n\tdie \"ERROR: use at most one of -p, -n, -L.\"\n(( opt_pid )) && ftext=\" issued by PID $pid\"\n(( opt_tid )) && ftext=\" issued by TID $tid\"\n(( opt_name )) && ftext=\" issued by process name \\\"$name\\\"\"\n(( opt_file )) && ftext=\"$ftext for filenames containing \\\"$file\\\"\"\nif (( opt_duration )); then\n\techo \"Tracing open()s$ftext for $duration seconds (buffered)...\"\nelse\n\techo \"Tracing open()s$ftext. Ctrl-C to end.\"\nfi\n\n### select awk\n(( opt_duration )) && use=mawk || use=gawk\t# workaround for mawk fflush()\n[[ -x /usr/bin/$use ]] && awk=$use || awk=awk\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and begin tracing\necho nop > current_tracer\nver=$(uname -r)\nif [[ \"$ver\" == 2.* || \"$ver\" == 3.[1-6].* ]]; then\n\t# rval is char *\n\tkprobe='r:getnameprobe getname +0($retval):string'\nelse\n\t# rval is struct filename *\n\tkprobe='r:getnameprobe getname +0(+0($retval)):string'\nfi\nif ! echo $kprobe >> kprobe_events; then\n\tedie \"ERROR: adding a kprobe for getname(). Exiting.\"\nfi\nif (( opt_pid )); then\n\tfilter=\n\tfor tid in /proc/$pid/task/*; do\n\t\tfilter=\"$filter || common_pid == ${tid##*/}\"\n\tdone\n\tfilter=${filter:3}  # trim leading ' || ' (four characters)\n\tif ! echo $filter > events/kprobes/getnameprobe/filter || \\\n\t    ! echo $filter > events/syscalls/sys_exit_open/filter || \\\n\t    ! echo $filter > events/syscalls/sys_exit_openat/filter\n\tthen\n\t    edie \"ERROR: setting -p $pid. Exiting.\"\n\tfi\nfi\nif (( opt_tid )); then\n\tif ! echo \"common_pid == $tid\" > events/kprobes/getnameprobe/filter || \\\n\t    ! echo \"common_pid == $tid\" > events/syscalls/sys_exit_open/filter || \\\n\t    ! echo \"common_pid == $tid\" > events/syscalls/sys_exit_openat/filter\n\tthen\n\t    edie \"ERROR: setting -L $tid. Exiting.\"\n\tfi\nfi\nif ! echo 1 > events/kprobes/getnameprobe/enable; then\n\tedie \"ERROR: enabling kprobe for getname(). Exiting.\"\nfi\nif ! echo 1 > events/syscalls/sys_exit_open/enable; then\n\tedie \"ERROR: enabling open() exit tracepoint. Exiting.\"\nfi\nif ! echo 1 > events/syscalls/sys_exit_openat/enable; then\n\tedie \"ERROR: enabling openat() exit tracepoint. Exiting.\"\nfi\n(( opt_time )) && printf \"%-16s \" \"TIMEs\"\nprintf \"%-16.16s %-6s %4s %s\\n\" \"COMM\" \"PID\" \"FD\" \"FILE\"\n\n#\n# Determine output format. It may be one of the following (newest first):\n#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION\n#           TASK-PID    CPU#    TIMESTAMP  FUNCTION\n# To differentiate between them, the number of header fields is counted,\n# and an offset set, to skip the extra column when needed.\n#\noffset=$($awk 'BEGIN { o = 0; }\n\t$1 == \"#\" && $2 ~ /TASK/ && NF == 6 { o = 1; }\n\t$2 ~ /TASK/ { print o; exit }' trace)\n\n### print trace buffer\nwarn \"echo > trace\"\n( if (( opt_duration )); then\n\t# wait then dump buffer\n\tsleep $duration\n\tcat trace\nelse\n\t# print buffer live\n\tcat trace_pipe\nfi ) | $awk -v o=$offset -v opt_name=$opt_name -v name=$name \\\n    -v opt_duration=$opt_duration -v opt_time=$opt_time -v opt_fail=$opt_fail \\\n    -v opt_file=$opt_file -v file=$file '\n\t# common fields\n\t$1 != \"#\" {\n\t\t# task name can contain dashes and space\n\t\tsplit($0, line, \"-\")\n\t\tsub(/^[ \\t\\r\\n]+/, \"\", line[1])\n\t\tcomm = line[1]\n\t\tif (opt_name && match(comm, name) == 0)\n\t\t\tnext\n\t\tsub(/ .*$/, \"\", line[2])\n\t\tpid = line[2]\n\t}\n\n\t# do_sys_open()\n\t$1 != \"#\" && $(5+o) ~ /do_sys_open/ {\n\t\t#\n\t\t# eg: ... (do_sys_open+0xc3/0x220 <- getname) arg1=\"file1\"\n\t\t#\n\t\tmatch($0, /arg1=\\\"(.+)\\\"/, m)\n\t\tlastfile[pid] = m[1]\n\t}\n\n\t# sys_open() / sys_openat()\n\t$1 != \"#\" && ($(4+o) == \"sys_open\" || $(4+o) == \"sys_openat\") {\n\t\tfilename = lastfile[pid]\n\t\tif (!filename)\n\t\t\tnext\n\t\tdelete lastfile[pid]\n\t\tif (opt_file && filename !~ file)\n\t\t\tnext\n\t\trval = $NF\n\t\t# matched failed as beginning with 0xfffff\n\t\tif (opt_fail && rval !~ /0xfffff/)\n\t\t\tnext\n\t\tif (rval ~ /0xfffff/)\n\t\t\trval = -1\n\n\t\tif (opt_time) {\n\t\t\ttime = $(3+o); sub(\":\", \"\", time)\n\t\t\tprintf \"%-16s \", time\n\t\t}\n\t\tprintf \"%-16.16s %-6s %4s %s\\n\", comm, pid, rval, filename\n\t}\n\n\t$0 ~ /LOST.*EVENTS/ {\n\t\tdelete lastfile\n\t\tprint \"WARNING: \" $0 > \"/dev/stderr\" }\n'\n\n### end tracing\nend\n"
  },
  {
    "path": "syscount",
    "content": "#!/bin/bash\n#\n# syscount - count system calls.\n#            Written using Linux perf_events (aka \"perf\").\n#\n# This is a proof-of-concept using perf_events capabilities for older kernel\n# versions, that lack custom in-kernel aggregations. Once they exist, this\n# script can be substantially rewritten and improved (lower overhead).\n#\n# USAGE: syscount [-chv] [-t top] {-p PID|-d seconds|command}\n#\n# Run \"syscount -h\" for full usage.\n#\n# REQUIREMENTS: Linux perf_events: add linux-tools-common, run \"perf\", then\n# add any additional packages it requests. Also needs awk.\n#\n# OVERHEADS: Modes that report syscall names only (-c, -cp PID, -cd secs) have\n# lower overhead, since they use in-kernel counts. Other modes which report\n# process IDs (-cv) or process names (default) create a perf.data file for\n# post processing, and you will see messages about it doing this. Beware of\n# the file size (test for short durations, or use -c to see counts based on\n# in-kernel counters), and gauge overheads based on the perf.data size.\n#\n# Note that this script delibrately does not pipe perf record into\n# perf script, which would avoid perf.data, because it can create a feedback\n# loop where the perf script syscalls are recorded. Hopefully there will be a\n# fix for this in a later perf version, so perf.data can be skipped, or other\n# kernel features to aggregate by process name in-kernel directly (eg, via\n# eBPF, ktap, or SystemTap).\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the syscount(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 07-Jul-2014\tBrendan Gregg\tCreated this.\n\n# default variables\nopt_count=0; opt_pid=0; opt_verbose=0; opt_cmd=0; opt_duration=0; opt_tail=0\ntnum=; pid=; duration=; cmd=; cpus=-a; opts=; tcmd=cat; ttext=\ntrap '' INT QUIT TERM PIPE HUP\n\nstdout_workaround=1\t# needed for older perf versions\nwrite_workaround=1\t# needed for perf versions that trace their own writes\n\n### parse options\nwhile getopts cd:hp:t:v opt\ndo\n\tcase $opt in\n\tc)\topt_count=1 ;;\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tt)\topt_tail=1; tnum=$OPTARG ;;\n\tv)\topt_verbose=1 ;;\n\th|?)\tcat <<-END >&2\n\t\tUSAGE: syscount [-chv] [-t top] {-p PID|-d seconds|command}\n\t\t       syscount                  # count by process name\n\t\t                -c               # show counts by syscall name\n\t\t                -h               # this usage message\n\t\t                -v               # verbose: shows PID\n\t\t                -p PID           # trace this PID only\n\t\t                -d seconds       # duration of trace\n\t\t                -t num           # show top number only\n\t\t                command          # run and trace this command\n\t\t  eg,\n\t\t        syscount                 # syscalls by process name\n\t\t        syscount -c              # syscalls by syscall name\n\t\t        syscount -d 5            # trace for 5 seconds\n\t\t        syscount -cp 923         # syscall names for PID 923\n\t\t        syscount -c ls           # syscall names for \"ls\"\n\n\t\tSee the man page and example file for more info.\n\t\tEND\n\t\texit 1\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### option logic\nif (( $# > 0 )); then\n\topt_cmd=1\n\tcmd=\"$@\"\n\tcpus=\nfi\nif (( opt_pid + opt_duration + opt_cmd > 1 )); then\n\techo >&2 \"ERROR: Pick one of {-p PID|-n name|-d seconds|command}\"\n\texit 1\nfi\nif (( opt_tail )); then\n\ttcmd=\"tail -$tnum\"\n\tttext=\" Top $tnum only.\"\nfi\nif (( opt_duration )); then\n\tcmd=\"sleep $duration\"\n\techo \"Tracing for $duration seconds.$ttext..\"\nfi\nif (( opt_pid )); then\n\tcpus=\n\tcmd=\"-p $pid\"\n\techo \"Tracing PID $pid.$ttext.. Ctrl-C to end.\"\nfi\n(( opt_cmd )) && echo \"Tracing while running: \\\"$cmd\\\".$ttext..\"\n(( opt_pid + opt_duration + opt_cmd == 0 )) && \\\n    echo \"Tracing.$ttext.. Ctrl-C to end.\"\n(( stdout_workaround )) && opts=\"-o /dev/stdout\"\n\nulimit -n 32768\t\t# often needed\n\n### execute syscall name mode\nif (( opt_count && ! opt_verbose )); then\n\t: ${cmd:=sleep 999999}\n\tout=$(perf stat $opts -e 'syscalls:sys_enter_*' $cpus $cmd)\n\tprintf \"%-17s %8s\\n\" \"SYSCALL\" \"COUNT\"\n\techo \"$out\" | awk '\n\t$1 && $2 ~ /syscalls:/ {\n\t\tsub(\"syscalls:sys_enter_\", \"\"); sub(\":\", \"\")\n\t\tgsub(\",\", \"\")\n\t\tprintf \"%-17s %8s\\n\", $2, $1\n\t}' | sort -n -k2 | $tcmd\n\texit\nfi\n\n### execute syscall name with pid mode\nif (( opt_count && opt_verbose )); then\n\tif (( write_workaround )); then\n\t\t# this list must end in write to associate the filter\n\t\ttp=$(perf list syscalls:sys_enter_* | awk '\n\t\t    $1 != \"syscalls:sys_enter_write\" &&  $1 ~ /syscalls:/ { printf \"-e %s \", $1 }')\n\t\ttp=\"$tp -e syscalls:sys_enter_write\"\n\t\tsh -c \"perf record $tp --filter 'common_pid != '\\$\\$ $cpus $cmd\"\n\telse\n\t\tperf record 'syscalls:sys_enter_*' $cpus $cmd\n\t\t# could also pipe direct to perf script\n\tfi\n\n\tprintf \"%-6s %-16s %-17s %8s\\n\" \"PID\" \"COMM\" \"SYSCALL\" \"COUNT\"\n\tperf script --fields pid,comm,event | awk '$1 != \"#\" {\n\t\tsub(\"syscalls:sys_enter_\", \"\"); sub(\":\", \"\")\n\t\ta[$1 \";\" $2 \";\" $3]++\n\t}\n\tEND {\n\t\tfor (k in a) {\n\t\t\tsplit(k, b, \";\");\n\t\t\tprintf \"%-6s %-16s %-17s %8d\\n\", b[2], b[1], b[3], a[k]\n\t\t}\n\t}' | sort -n -k4 | $tcmd\n\texit\nfi\n\n### execute process name mode\ntp=\"-e raw_syscalls:sys_enter\"\nif (( write_workaround )); then\n\tsh -c \"perf record $tp --filter 'common_pid != '\\$\\$ $cpus $cmd\"\nelse\n\tperf record $tp $cpus $cmd\nfi\n\nif (( opt_verbose )); then\n\tprintf \"%-6s %-16s %8s\\n\" \"PID\" \"COMM\" \"COUNT\"\n\tperf script --fields pid,comm | awk '$1 != \"#\" { a[$1 \";\" $2]++ }\n\tEND {\n\t\tfor (k in a) {\n\t\t\tsplit(k, b, \";\");\n\t\t\tprintf \"%-6s %-16s %8d\\n\", b[2], b[1],  a[k]\n\t\t}\n\t}' | sort -n -k3 | $tcmd\nelse\n\tprintf \"%-16s %8s\\n\" \"COMM\" \"COUNT\"\n\tperf script --fields comm | awk '$1 != \"#\" { a[$1]++ }\n\tEND {\n\t\tfor (k in a) {\n\t\t\tprintf \"%-16s %8d\\n\", k,  a[k]\n\t\t}\n\t}' | sort -n -k2 | $tcmd\nfi\n"
  },
  {
    "path": "system/tpoint",
    "content": "#!/bin/bash\n#\n# tpoint - trace a given tracepoint. Static tracing.\n#          Written using Linux ftrace.\n#\n# This will enable a given tracepoint, print events, then disable the tracepoint\n# when the program ends. This is like a simple version of the \"perf\" command for\n# printing live tracepoint events only. Wildcards are currently not supported.\n# If this is insufficient for any reason, use the perf command instead.\n#\n# USAGE: ./tpoint [-hHsv] [-d secs] [-p pid] [-L tid] tracepoint [filter]\n#        ./tpoint -l\n#\n# Run \"tpoint -h\" for full usage.\n#\n# I wrote this because I often needed a quick way to dump stack traces for a\n# given tracepoint.\n#\n# OVERHEADS: Relative to the frequency of traced events. You might want to\n# check their frequency beforehand, using perf_events. Eg:\n#\n#\tperf stat -e block:block_rq_issue -a sleep 5\n#\n# To count occurrences of that tracepoint for 5 seconds.\n#\n# REQUIREMENTS: FTRACE and tracepoints, which you may already have on recent\n# kernel versions.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the tpoint(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 22-Jul-2014\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock; wroteflock=0\nopt_duration=0; duration=; opt_pid=0; pid=; opt_tid=0; tid=\nopt_filter=0; filter=; opt_view=0; opt_headers=0; opt_stack=0; dmesg=2\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: tpoint [-hHsv] [-d secs] [-p PID] [-L TID] tracepoint [filter]\n\t       tpoint -l\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -p PID          # PID to match on event\n\t                 -L TID          # thread id to match on event\n\t                 -v              # view format file (don't trace)\n\t                 -H              # include column headers\n\t                 -l              # list all tracepoints\n\t                 -s              # show kernel stack traces\n\t                 -h              # this usage message\n\t   eg,\n\t       tpoint -l | grep open\n\t                                 # find tracepoints containing \"open\"\n\t       tpoint syscalls:sys_enter_open\n\t                                 # trace open() syscall entry\n\t       tpoint block:block_rq_issue\n\t                                 # trace block I/O issue\n\t       tpoint -s block:block_rq_issue\n\t                                 # show kernel stacks\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > $tdir/enable\"\n\tif (( opt_filter )); then\n\t\twarn \"echo 0 > $tdir/filter\"\n\tfi\n\t(( opt_stack )) && warn \"echo 0 > options/stacktrace\"\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\n### process options\nwhile getopts d:hHlp:L:sv opt\ndo\n\tcase $opt in\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tL)\topt_tid=1; tid=$OPTARG ;;\n\tH)\topt_headers=1 ;;\n\tl)\topt_list=1 ;;\n\ts)\topt_stack=1 ;;\n\tv)\topt_view=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nif (( !opt_list )); then\n\tshift $(( $OPTIND - 1 ))\n\t(( $# )) || usage\n\ttpoint=$1\n\tshift\n\tif (( $# )); then\n\t\topt_filter=1\n\t\tfilter=$1\n\tfi\nfi\n\n### option logic\n(( opt_pid + opt_filter + opt_tid > 1 )) && \\\n\tdie \"ERROR: use at most one of -p, -L, or filter.\"\n(( opt_duration && opt_view )) && die \"ERROR: use either -d or -v.\"\nif (( opt_pid )); then\n\t# convert to filter\n\topt_filter=1\n\t# ftrace common_pid is thread id from user's perspective\n\tfor tid in /proc/$pid/task/*; do\n\t\tfilter=\"$filter || common_pid == ${tid##*/}\"\n\tdone\n\tfilter=${filter:3}  # trim leading ' || ' (four characters)\nfi\nif (( opt_tid )); then\n\topt_filter=1\n\tfilter=\"common_pid == $tid\"\nfi\nif (( !opt_view && !opt_list )); then\n\tif (( opt_duration )); then\n\t\techo \"Tracing $tpoint for $duration seconds (buffered)...\"\n\telse\n\t\techo \"Tracing $tpoint. Ctrl-C to end.\"\n\tfi\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\n### do list tracepoints\nif (( opt_list )); then\n\tcd events\n\tfor tp in */*; do\n\t\t# skip filter/enable files\n\t\t[[ -f $tp ]] && continue\n\t\techo ${tp/\\//:}\n\tdone\n\texit\nfi\n\n### check tracepoints\ntdir=events/${tpoint/:/\\/}\n[[ -e $tdir ]] || die \"ERROR: tracepoint $tpoint not found. Exiting\"\n\n### view\nif (( opt_view )); then\n\tcat $tdir/format\n\texit\nfi\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and begin tracing\necho nop > current_tracer\nif (( opt_filter )); then\n\tif ! echo \"$filter\" > $tdir/filter; then\n\t\tedie \"ERROR: setting filter or -p. Exiting.\"\n\tfi\nfi\nif (( opt_stack )); then\n\tif ! echo 1 > options/stacktrace; then\n\t\tedie \"ERROR: enabling stack traces (-s). Exiting\"\n\tfi\nfi\nif ! echo 1 >> $tdir/enable; then\n\tedie \"ERROR: enabling tracepoint $tprobe. Exiting.\"\nfi\n\n### print trace buffer\nwarn \"echo > trace\"\nif (( opt_duration )); then\n\tsleep $duration\n\tif (( opt_headers )); then\n\t\tcat trace\n\telse\n\t\tgrep -v '^#' trace\n\tfi\nelse\n\t# trace_pipe lack headers, so fetch them from trace\n\t(( opt_headers )) && cat trace\n\tcat trace_pipe\nfi\n\n### end tracing\nend\n"
  },
  {
    "path": "tools/reset-ftrace",
    "content": "#!/bin/bash\n#\n# reset-ftrace - reset state of ftrace, disabling all tracing.\n#                Written for Linux ftrace.\n#\n# This may only be of use to ftrace hackers who, in the process of developing\n# ftrace software, often get the subsystem into a partially active state, and\n# would like a quick way to reset state. Check the end of this script for the\n# actually files reset, and add more if you need.\n#\n# USAGE: ./reset-ftrace [-fhq]\n#\n# REQUIREMENTS: FTRACE CONFIG.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the reset-ftrace(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 20-Jul-2014\tBrendan Gregg\tCreated this.\n\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock\nopt_force=0; opt_quiet=0\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: reset-ftrace [-fhq]\n\t                 -f              # force: delete ftrace lock file\n\t                 -q              # quiet: reset, but say nothing\n\t                 -h              # this usage message\n\t  eg,\n\t       reset-ftrace              # disable active ftrace session\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction vecho {\n\t(( opt_quiet )) && return\n\techo \"$@\"\n}\n\n# write to file\nfunction writefile {\n\tfile=$1\n\tstring=$2\t# optional\n\tif [[ ! -w $file ]]; then\n\t\techo >&2 \"WARNING: file $file not writable/exists. Skipping.\"\n\t\treturn\n\tfi\n\tvecho \"$file, before:\"\n\t(( ! opt_quiet )) && cat -n $file\n\twarn \"echo $string > $file\"\n\tvecho \"$file, after:\"\n\t(( ! opt_quiet )) && cat -n $file\n\tvecho\n}\n\n### process options\nwhile getopts fhq opt\ndo\n\tcase $opt in\n\tf)\topt_force=1 ;;\n\tq)\topt_quiet=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\n\n### ftrace lock\nif [[ -e $flock ]]; then\n\tif (( opt_force )); then\n\t\twarn rm $flock\n\telse\n\t\techo -e >&2 \"ERROR: ftrace lock ($flock) exists. It shows\" \\\n\t\t    \"ftrace may be in use by PID $(cat $flock).\\nDouble check\" \\\n\t\t    \"to see if that PID is still active. If not, consider\" \\\n\t\t    \"using -f to force a reset. Exiting.\"\n\t\texit 1\n\tfi\nfi\n\n### reset ftrace state\nvecho \"Reseting ftrace state...\"\nvecho\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\"\nwritefile current_tracer nop\nwritefile set_ftrace_filter\nwritefile set_graph_function\nwritefile set_ftrace_pid\nwritefile events/enable 0\nwritefile tracing_thresh 0\nwritefile kprobe_events\nwritefile tracing_on 1\nvecho \"Done.\"\n"
  },
  {
    "path": "user/uprobe",
    "content": "#!/bin/bash\n#\n# uprobe - trace a given uprobe definition. User-level dynamic tracing.\n#          Written using Linux ftrace. Experimental.\n#\n# This will create, trace, then destroy a given uprobe definition. See\n# Documentation/trace/uprobetrace.txt in the Linux kernel source for the\n# syntax of a uprobe definition, and \"uprobe -h\" for examples. With this tool,\n# the probe alias is optional (it will default to something meaningful).\n#\n# USAGE: ./uprobe [-FhHsv] [-d secs] [-p pid] [-L tid] {-l target |\n#                 uprobe_definition [filter]}\n#\n# Run \"uprobe -h\" for full usage.\n#\n# WARNING: This uses dynamic tracing of user-level functions, using some\n# relatively new kernel code. I have seen this cause target processes to fail,\n# either entering endless spin loops or crashing on illegal instructions. I\n# believe newer kernels (post 4.0) are relatively safer, but use caution. Test\n# in a lab environment, and know what you are doing, before use.\n#\n# Use extreme caution with the raw address mode: eg, \"p:libc:0xbf130\". uprobe\n# does not check for instruction alignment, so tracing the wrong address (eg,\n# mid-way through a multi-byte instruction) will corrupt the target's memory.\n# Other tracers (eg, perf_events with debuginfo) check alignment.\n#\n# Also beware of widespread tracing that interferes with the operation of the\n# system, eg, tracing libc:malloc, which by-default will trace _all_ processes.\n# Test in a lab environment before use.\n#\n# I wrote this because I kept testing different custom uprobes at the command\n# line, and wanted a way to automate the steps. For generic user-level\n# tracing, use perf_events directly.\n#\n# REQUIREMENTS: FTRACE and UPROBE CONFIG, which you may already have on recent\n# kernel versions, file(1), ldconfig(8), objdump(1), and some version of awk.\n# Also, currently only executes on Linux 4.0+ (see WARNING) unless -F is used.\n#\n# From perf-tools: https://github.com/brendangregg/perf-tools\n#\n# See the uprobe(8) man page (in perf-tools) for more info.\n#\n# COPYRIGHT: Copyright (c) 2015 Brendan Gregg.\n#\n#  This program is free software; you can redistribute it and/or\n#  modify it under the terms of the GNU General Public License\n#  as published by the Free Software Foundation; either version 2\n#  of the License, or (at your option) any later version.\n#\n#  This program is distributed in the hope that it will be useful,\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n#  GNU General Public License for more details.\n#\n#  You should have received a copy of the GNU General Public License\n#  along with this program; if not, write to the Free Software Foundation,\n#  Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.\n#\n#  (http://www.gnu.org/copyleft/gpl.html)\n#\n# 27-Jul-2015\tBrendan Gregg\tCreated this.\n\n### default variables\ntracing=/sys/kernel/debug/tracing\nflock=/var/tmp/.ftrace-lock; wroteflock=0\nopt_duration=0; duration=; opt_pid=0; pid=; opt_tid=0; tid=\nopt_filter=0; filter=; opt_view=0; opt_headers=0; opt_stack=0; dmesg=2\ndebug=0; opt_force=0; opt_list=0; target=\nPATH=$PATH:/usr/bin:/sbin\t# ensure we find objdump, ldconfig\ntrap ':' INT QUIT TERM PIPE HUP\t# sends execution to end tracing section\n\nfunction usage {\n\tcat <<-END >&2\n\tUSAGE: uprobe [-FhHsv] [-d secs] [-p PID] [-L TID] {-l target |\n\t              uprobe_definition [filter]}\n\t                 -F              # force. trace despite warnings.\n\t                 -d seconds      # trace duration, and use buffers\n\t                 -l target       # list functions from this executable\n\t                 -p PID          # PID to match on events\n\t                 -L TID          # thread id to match on events\n\t                 -v              # view format file (don't trace)\n\t                 -H              # include column headers\n\t                 -s              # show user stack traces\n\t                 -h              # this usage message\n\t\n\tNote that these examples may need modification to match your kernel\n\tversion's function names and platform's register usage.\n\t   eg,\n\t       # trace readline() calls in all running \"bash\" executables:\n\t           uprobe p:bash:readline\n\t       # trace readline() with explicit executable path:\n\t           uprobe p:/bin/bash:readline\n\t       # trace the return of readline() with return value as a string:\n\t           uprobe 'r:bash:readline +0(\\$retval):string'\n\t       # trace sleep() calls in all running libc shared libraries:\n\t           uprobe p:libc:sleep\n\t       # trace sleep() with register %di (x86):\n\t           uprobe 'p:libc:sleep %di'\n\t       # trace this address (use caution: must be instruction aligned):\n\t           uprobe p:libc:0xbf130\n\t       # trace gettimeofday() for PID 1182 only:\n\t           uprobe -p 1182 p:libc:gettimeofday\n\t       # trace the return of fopen() only when it returns NULL:\n\t           uprobe 'r:libc:fopen file=\\$retval' 'file == 0'\n\n\tSee the man page and example file for more info.\nEND\n\texit\n}\n\nfunction warn {\n\tif ! eval \"$@\"; then\n\t\techo >&2 \"WARNING: command failed \\\"$@\\\"\"\n\tfi\n}\n\nfunction end {\n\t# disable tracing\n\techo 2>/dev/null\n\techo \"Ending tracing...\" 2>/dev/null\n\tcd $tracing\n\twarn \"echo 0 > events/uprobes/$uname/enable\"\n\tif (( opt_filter )); then\n\t\twarn \"echo 0 > events/uprobes/$uname/filter\"\n\tfi\n\twarn \"echo -:$uname >> uprobe_events\"\n\t(( opt_stack )) && warn \"echo 0 > options/userstacktrace\"\n\twarn \"echo > trace\"\n\t(( wroteflock )) && warn \"rm $flock\"\n}\n\nfunction die {\n\techo >&2 \"$@\"\n\texit 1\n}\n\nfunction edie {\n\t# die with a quiet end()\n\techo >&2 \"$@\"\n\texec >/dev/null 2>&1\n\tend\n\texit 1\n}\n\nfunction set_path {\n\tname=$1\n\n\tpath=$(which $name)\n\tif [[ \"$path\" == \"\" ]]; then\n\t\tpath=$(ldconfig -v 2>/dev/null | awk -v lib=$name '\n\t\t    $1 ~ /:/ { sub(/:/, \"\", $1); path = $1 }\n\t\t    { sub(/\\..*/, \"\", $1); }\n\t\t    $1 == lib { print path \"/\" $3 }')\n\t\tif [[ \"$path\" == \"\" ]]; then\n\t\t\tdie \"ERROR: segment \\\"$name\\\" ambiguous.\" \\\n\t\t\t    \"Program or library? Try a full path.\"\n\t\tfi\n\tfi\n\n\tif [[ ! -x $path ]]; then\n\t\tdie \"ERROR: resolved \\\"$name\\\" to \\\"$path\\\", but file missing\"\n\tfi\n}\n\nfunction set_addr {\n\tpath=$1\n\tname=$2\n\tsym=$3\n\n\t[[ \"$path\" == \"\" ]] && die \"ERROR: missing symbol path.\"\n\t[[ \"$sym\" == \"\" ]] && die \"ERROR: missing symbol for $path\"\n\n\taddr=$(objdump -tT $path | awk -v sym=$sym '\n\t    $NF == sym && $4 == \".text\"  { print $1; exit }')\n\t[[ \"$addr\" == \"\" ]] && die \"ERROR: missing symbol \\\"$sym\\\" in $path\"\n\t(( 0x$addr == 0 )) && die \"ERROR: failed resolving \\\"$sym\\\" in $path.\" \\\n\t    \"Maybe it exists in a different target (eg, library)?\"\n\taddr=0x$( printf \"%x\" 0x$addr )\t\t# strip leading zeros\n\n\ttype=$(file $path)\n\tif [[ \"$type\" != *shared?object* ]]; then\n\t\t# subtract the base mapping address. see Documentation/trace/\n\t\t# uprobetracer.txt for background.\n\t\tbase=$(objdump -x $path | awk '\n\t\t    $1 == \"LOAD\" && $3 ~ /^[0x]*$/ { print $5 }')\n\t\t[[ \"$base\" != 0x* ]] && die \"ERROR: finding base load addr\"\\\n\t\t    \"for $path.\"\n\t\taddr=$(( addr - base ))\n\t\t(( addr < 0 )) && die \"ERROR: transposed address for $sym\"\\\n\t\t    \"became negative: $addr\"\n\t\taddr=0x$( printf \"%x\" $addr)\n\tfi\n}\n\n### process options\nwhile getopts Fd:hHl:p:L:sv opt\ndo\n\tcase $opt in\n\tF)\topt_force=1 ;;\n\td)\topt_duration=1; duration=$OPTARG ;;\n\tp)\topt_pid=1; pid=$OPTARG ;;\n\tL)\topt_tid=1; tid=$OPTARG ;;\n\tl)\topt_list=1; target=$OPTARG ;;\n\tH)\topt_headers=1 ;;\n\ts)\topt_stack=1 ;;\n\tv)\topt_view=1 ;;\n\th|?)\tusage ;;\n\tesac\ndone\nshift $(( $OPTIND - 1 ))\nuprobe=$1\nshift\nif (( $# )); then\n\topt_filter=1\n\tfilter=$1\nfi\n\n### handle listing\n[[ \"$opt_list\" == 1 && \"$uprobe\" != \"\" ]] && die \"ERROR: -l takes a target only\"\nif (( opt_list )); then\n\tif [[ \"$target\" != */* ]]; then\n\t\tset_path $target\n\t\ttarget=$path\n\tfi\n\tobjdump -tT $target | awk '$4 == \".text\" { print $NF }' | sort | uniq\n\texit\nfi\n\n### check kernel version\nver=$(uname -r)\nmaj=${ver%%.*}\nif (( opt_force == 0 && $maj < 4 )); then\n\tcat <<-END >&2\n\tERROR: Kernel version >= 4.0 preferred (you have $ver). Aborting.\n\t\n\tBackground: uprobes were first added in 3.5. I've tested them on 3.13,\n\tand found them unsafe, as they can crash or lock up processes, which can\n\teffectively lock up the system. On 4.0, uprobes seem much safer. You\n\tcan use -F to force tracing, but you've been warned.\nEND\n\texit\nfi\n\n### check command dependencies\nfor cmd in file objdump ldconfig awk; do\n\twhich $cmd > /dev/null\n\t(( $? != 0 )) && die \"ERROR: missing $cmd in \\$PATH. $0 needs\" \\\n\t    \"to use this command. Exiting.\"\ndone\n\n### option logic\n[[ \"$uprobe\" == \"\" ]] && usage\n(( opt_pid + opt_filter + opt_tid > 1 )) && \\\n\tdie \"ERROR: use at most one of -p, -L, or filter.\"\n(( opt_duration && opt_view )) && die \"ERROR: use either -d or -v.\"\nif (( opt_pid )); then\n\t# convert to filter\n\topt_filter=1\n\t# ftrace common_pid is thread id from user's perspective\n\tfor tid in /proc/$pid/task/*; do\n\t\tfilter=\"$filter || common_pid == ${tid##*/}\"\n\tdone\n\tfilter=${filter:3}  # trim leading ' || ' (four characters)\nfi\nif (( opt_tid )); then\n\topt_filter=1\n\tfilter=\"common_pid == $tid\"\nfi\nif [[ \"$uprobe\" != p:* && \"$uprobe\" != r:* ]]; then\n\techo >&2 \"ERROR: invalid uprobe definition (should start with p: or r:)\"\n\tusage\nfi\n#\n# Parse the following:\n# p:bash:readline\n# p:my bash:readline\n# p:bash:readline %si\n# r:bash:readline $ret\n# p:my bash:readline %si\n# p:bash:readline si=%si\n# p:my bash:readline si=%si\n# r:bash:readline cmd=+0($retval):string\n# ... and all of the above with /bin/bash instead of bash\n# ... and all of the above with libc:sleep instead of ...\n# ... and all of the above with /lib/x86_64-linux-gnu/libc.so.6:sleep ...\n# ... and all of the above with symbol addresses\n# ... and examples from USAGE message\n# The following code is not as complicated as it looks.\n#\nutype=${uprobe%%:*}\nurest=\"${uprobe#*:} \"\nset -- $urest\nif [[ $1 == *:* ]]; then\n\tuname=; probe=$1; shift; uargs=\"$@\"\nelse\n\t[[ $2 != *:* ]] && die \"ERROR: invalid probe. See usage (-h).\"\n\tuname=$1; probe=$2; shift 2; uargs=\"$@\"\nfi\npath=$probe; path=${path%%:*}\naddr=$probe; addr=${addr##*:}\n\n# set seg and fix path (eg, seg=bash, path=/bin/bash)\nif [[ $path == */* ]]; then\n\tseg=${path##*/}\n\tseg=${seg%%.*}\nelse\n\tseg=$path\n\t# determine path, eg, given \"zsh\" or \"libc\"\n\tset_path $path\nfi\n\n# fix uname and addr (eg, uname=readline, addr=0x8db60)\nif [[ \"$addr\" == 0x* ]]; then\n\t# symbol unknown; default to seg+addr\n\t[[ \"$uname\" == \"\" ]] && uname=${seg}_$addr\nelse\n\t[[ \"$uname\" == \"\" ]] && uname=$addr\n\tset_addr $path $seg $addr\nfi\n\n# construct uprobe\nuprobe=\"$utype:$uname $path:$addr\"\n[[ \"$uargs\" != \"\" ]] && uprobe=\"$uprobe $uargs\"\n\nif (( debug )); then\n\techo \"uname: \\\"$uname\\\", uprobe: \\\"$uprobe\\\"\"\nfi\n\n### check permissions\ncd $tracing || die \"ERROR: accessing tracing. Root user? Kernel has FTRACE?\n    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)\"\n\nif (( !opt_view )); then\n\tif (( opt_duration )); then\n\t\techo \"Tracing uprobe $uname for $duration seconds (buffered)...\"\n\telse\n\t\techo \"Tracing uprobe $uname ($uprobe). Ctrl-C to end.\"\n\tfi\nfi\n\n### ftrace lock\n[[ -e $flock ]] && die \"ERROR: ftrace may be in use by PID $(cat $flock) $flock\"\necho $$ > $flock || die \"ERROR: unable to write $flock.\"\nwroteflock=1\n\n### setup and begin tracing\necho nop > current_tracer\nif ! echo \"$uprobe\" >> uprobe_events; then\n\techo >&2 \"ERROR: adding uprobe \\\"$uprobe\\\".\"\n\tif (( dmesg )); then\n\t\techo >&2 \"Last $dmesg dmesg entries (might contain reason):\"\n\t\tdmesg | tail -$dmesg | sed 's/^/    /'\n\tfi\n\tedie \"Exiting.\"\nfi\nif (( opt_view )); then\n\tcat events/uprobes/$uname/format\n\tedie \"\"\nfi\nif (( opt_filter )); then\n\tif ! echo \"$filter\" > events/uprobes/$uname/filter; then\n\t\tedie \"ERROR: setting filter or -p. Exiting.\"\n\tfi\nfi\nif (( opt_stack )); then\n\tif ! echo 1 > options/userstacktrace; then\n\t\tedie \"ERROR: enabling stack traces (-s). Exiting\"\n\tfi\nfi\nif ! echo 1 > events/uprobes/$uname/enable; then\n\tedie \"ERROR: enabling uprobe $uname. Exiting.\"\nfi\n\n### print trace buffer\nwarn \"echo > trace\"\nif (( opt_duration )); then\n\tsleep $duration\n\tif (( opt_headers )); then\n\t\tcat trace\n\telse\n\t\tgrep -v '^#' trace\n\tfi\nelse\n\t# trace_pipe lack headers, so fetch them from trace\n\t(( opt_headers )) && cat trace\n\tcat trace_pipe\nfi\n\n### end tracing\nend\n"
  }
]