Showing preview only (334K chars total). Download the full file or copy to clipboard to get everything.
Repository: lowleveldesign/debug-recipes
Branch: main
Commit: 2a6278c1c018
Files: 36
Total size: 320.4 KB
Directory structure:
gitextract_r48gq6i_/
├── .gitignore
├── 404.html
├── CNAME
├── Gemfile
├── LICENSE
├── README.md
├── _config.yml
├── _includes/
│ ├── footer.html
│ └── head.html
├── _layouts/
│ ├── home.html
│ └── posts.html
├── about.md
├── articles.md
├── assets/
│ ├── main.scss
│ └── other/
│ ├── EtwMetadata.ps1.txt
│ ├── WTComTrace.wprp
│ ├── winapi-user32.ps1.txt
│ └── windbg-install.ps1.txt
├── browserconfig.xml
├── guides/
│ ├── com-troubleshooting.md
│ ├── configuring-linux-for-effective-troubleshooting.md
│ ├── configuring-windows-for-effective-troubleshooting.md
│ ├── diagnosing-dotnet-apps.md
│ ├── diagnosing-native-windows-apps.md
│ ├── ebpf.md
│ ├── etw.md
│ ├── gdb.md
│ ├── linux-tracing.md
│ ├── network-tracing-tools.md
│ ├── using-withdll-and-detours-to-trace-winapi.md
│ ├── windbg.md
│ └── windows-performance-counters.md
├── guides.md
├── index.md
├── site.webmanifest
└── tools.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
_site
.sass-cache
.jekyll-cache
.jekyll-metadata
vendor
draft_*
================================================
FILE: 404.html
================================================
---
permalink: /404.html
layout: default
---
<style type="text/css" media="screen">
.container {
margin: 10px auto;
max-width: 600px;
text-align: center;
}
h1 {
margin: 30px 0;
font-size: 4em;
line-height: 1;
letter-spacing: -1px;
}
</style>
<div class="container">
<h1>404</h1>
<p><strong>Page not found :(</strong></p>
<p>The requested page could not be found.</p>
</div>
================================================
FILE: CNAME
================================================
wtrace.net
================================================
FILE: Gemfile
================================================
source "https://rubygems.org"
# Hello! This is where you manage which Jekyll version is used to run.
# When you want to use a different version, change it below, save the
# file and run `bundle install`. Run Jekyll with `bundle exec`, like so:
#
# bundle exec jekyll serve
#
# This will help ensure the proper Jekyll version is running.
# Happy Jekylling!
# gem "jekyll", "~> 4.2.0"
# This is the default theme for new Jekyll sites. You may change this to anything you like.
gem "minima", "~> 2.5"
# gem "jekyll-theme-cayman", "~> 0.2.0"
# If you want to use GitHub Pages, remove the "gem "jekyll"" above and
# uncomment the line below. To upgrade, run `bundle update github-pages`.
gem "github-pages", group: :jekyll_plugins
# If you have any plugins, put them here!
group :jekyll_plugins do
gem "jekyll-feed", "~> 0.12"
end
# Windows and JRuby does not include zoneinfo files, so bundle the tzinfo-data gem
# and associated library.
platforms :mingw, :x64_mingw, :mswin, :jruby do
gem "tzinfo", "~> 1.2"
gem "tzinfo-data"
end
# Performance-booster for watching directories on Windows
gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin]
gem "webrick", "~> 1.7"
gem "json", "~> 2.7"
================================================
FILE: LICENSE
================================================
Attribution 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution 4.0 International Public License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution 4.0 International Public License ("Public License"). To the
extent this Public License may be interpreted as a contract, You are
granted the Licensed Rights in consideration of Your acceptance of
these terms and conditions, and the Licensor grants You such rights in
consideration of benefits the Licensor receives from making the
Licensed Material available under these terms and conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
d. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
e. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
f. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
g. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
h. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
i. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
j. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
k. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part; and
b. produce, reproduce, and Share Adapted Material.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
4. If You Share Adapted Material You produce, the Adapter's
License You apply must not prevent recipients of the Adapted
Material from complying with this Public License.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material; and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.
================================================
FILE: README.md
================================================
Debug Recipes
=============
It is a repository of my field notes collected while debugging various .NET application problems on Windows (mainly) and Linux. They do not contain much theory but rather describe tools and scripts with some usage examples.
:floppy_disk: Old and no longer updated recipes are in the [archived branch](https://github.com/lowleveldesign/debug-recipes/tree/archive).
The recipes are available in the guides folder and at **[wtrace.net](https://wtrace.net/guides)** (probably the best way to view them).
## Troubleshooting guides
- [Diagnosing .NET applications](guides/diagnosing-dotnet-apps.md)
- [Diagnosing native Windows applications](guides/diagnosing-native-windows-apps.md)
- [COM troubleshooting](guides/com-troubleshooting)
## Tools usage guides
- [WinDbg usage guide](guides/windbg.md)
- [Event Tracing for Windows (ETW)](guides/etw.md)
- [Using withdll and detours to trace Win API calls](guides/using-withdll-and-detours-to-trace-winapi.md)
- [Windows Performance Counters](guides/windows-performance-counters.md)
- [Network tracing tools](guides/network-tracing-tools.md)
================================================
FILE: _config.yml
================================================
title: wtrace.net
email: contact@wtrace.net
description: >- # this means to ignore newlines until "baseurl:"
Tools and materials for software and system troubleshooting
baseurl: "" # the subpath of your site, e.g. /blog
url: "https://wtrace.net" # the base hostname & protocol for your site, e.g. http://example.com
youtube_username: "@lowleveldesign"
github_username: lowleveldesign
permalink: pretty
defaults:
-
scope:
path: ""
type: "posts"
values:
permalink: /:year/:month/:day/:title
# Build settings
theme: minima
plugins:
- jekyll-feed
- jekyll-seo-tag
- jekyll-redirect-from
- jekyll-sitemap
- jemoji
header_pages:
- guides.md
- tools.md
- about.md
================================================
FILE: _includes/footer.html
================================================
<footer class="site-footer h-card">
<data class="u-url" href="{{ "/" | relative_url }}"></data>
<div class="wrapper">
<div class="footer-col-wrapper">
<div class="footer-col footer-col-1">
<p>
© {{ site.time | date: "%Y" }} {{ site.title | escape }}
</p>
<p>{{- site.description | escape -}}</p>
</div>
<div class="footer-col footer-col-2">
<ul class="contact-list">
</ul>
</div>
<div class="footer-col footer-col-3">
{%- include social.html -%}
</div>
</div>
</div>
</footer>
================================================
FILE: _includes/head.html
================================================
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
<link rel="manifest" href="/site.webmanifest">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
{%- seo -%}
<link rel="stylesheet" href="{{ "/assets/main.css" | relative_url }}">
{%- feed_meta -%}
{%- if jekyll.environment == 'production' and site.google_analytics -%}
{%- include google-analytics.html -%}
{%- endif -%}
</head>
================================================
FILE: _layouts/home.html
================================================
---
---
<!DOCTYPE html>
<html lang="{{ page.lang | default: site.lang | default: "en" }}">
{%- include head.html -%}
<body>
{%- include header.html -%}
<div class="home">
{%- if page.title -%}
<div class="feature-image"{% if page.feature_image %} style="background-image: url({{ page.feature_image }})"{% endif %}>
<div class="wrapper">
<h1 class="page-heading">{{ page.title }}</h1>
{% if page.description %}
<p>{{ page.description }}</p>
{% endif %}
</div>
</div>
{%- endif -%}
<main class="page-content" aria-label="Content">
<div class="wrapper">
{{ content }}
</div>
</main>
</div>
{%- include footer.html -%}
</body>
</html>
================================================
FILE: _layouts/posts.html
================================================
---
layout: default
---
<div class="home">
{%- if page.title -%}
<h1 class="page-heading">{{ page.title }}</h1>
{%- endif -%}
{%- if site.posts.size > 0 -%}
<ul class="post-list">
{%- for post in site.posts -%}
<li>
{%- assign date_format = site.minima.date_format | default: "%b %-d, %Y" -%}
<span class="post-meta">{{ post.date | date: date_format }}</span>
<h3>
<a class="post-link" href="{{ post.url | relative_url }}">
{{ post.title | escape }}
</a>
</h3>
{{ post.excerpt }}
</li>
{%- endfor -%}
</ul>
{%- endif -%}
</div>
================================================
FILE: about.md
================================================
---
layout: page
title: About
---
I am **Sebastian Solnica**, a software engineer with more than 15 years of experience. My primary interests are debugging, profiling, and application security. I created this website to share tools and resources that can help you in your diagnostic endeavors.
I also provide consulting services for troubleshooting .NET applications. If you would like to discuss consulting or contact me for any other reason, please use [the contact form on my blog](https://lowleveldesign.org/about/) or email me at contact@wtrace.net.
<p class="credits">
<em><strong>Credits:</strong> this site uses modified icons from the <a href="https://github.com/feathericons/feather">feather set</a>.</em>
</p>
<p class="credits">
<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png"></a><br>The published guides are licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
</p>
================================================
FILE: articles.md
================================================
---
layout: page
title: Articles
redirect_to: /guides
---
================================================
FILE: assets/main.scss
================================================
---
# Only the main Sass file needs front matter (the dashes are enough)
---
$brand-color: #CA4E07;
$credits-color: #707070;
@import "minima";
body {
background-color: #f6f6ef;
}
pre, code {
background: transparent;
}
.highlighter-rouge .highlight {
background: #f9f9f9;
}
.highlight .c {
color: #6c6c62;
}
.post-title {
@include relative-font-size(2.2);
letter-spacing: -1px;
line-height: 1;
@include media-query($on-laptop) {
@include relative-font-size(2.0);
}
}
.post-content {
table {
table-layout: fixed;
}
table th {
text-align: center;
}
table td {
vertical-align: top;
}
h2, h3 {
margin: 15px 0 15px 0;
}
}
.site-title {
@include relative-font-size(1.4);
font-weight: 700;
line-height: $base-line-height * $base-font-size * 2.25;
letter-spacing: -1px;
margin-bottom: 0;
float: left;
text-transform: uppercase;
&, &:visited {
color: $brand-color;
}
}
.site-nav {
.page-link {
text-transform: uppercase;
font-weight: 600;
}
}
.feature-image {
background-color: black;
background-repeat: no-repeat;
margin-bottom: 10px;
padding-top: 50px;
height: 300px;
.wrapper {
color: #ffffff;
h1 {
font-size: 4rem;
font-weight: 900;
margin-bottom: 0px
}
p {
font-size: 1.2rem;
}
}
}
p.credits {
color: $credits-color;
padding-top: 10px;
margin-top: 10px;
}
================================================
FILE: assets/other/EtwMetadata.ps1.txt
================================================
$ErrorActionPreference = "Stop"
$MetadataFolder = "$env:LOCALAPPDATA\MyEtwMetadata\ById"
$MetadataSearchByNameFolder = "$env:LOCALAPPDATA\MyEtwMetadata\ByName"
if (-not (Test-Path $MetadataFolder)) {
New-Item -ItemType Directory -Path $MetadataFolder | Out-Null
}
if (-not (Test-Path $MetadataSearchByNameFolder)) {
New-Item -ItemType Directory -Path $MetadataSearchByNameFolder | Out-Null
}
function _SanitizeFileName {
param ([Parameter(Mandatory = $true)]$FileName)
[System.IO.Path]::GetInvalidFileNameChars() | ForEach-Object -Process {
$FileName = $FileName.Replace($_, [char]'_')
}
$FileName
}
Write-Output "Initializing ETW providers metadata... "
wevtutil.exe ep | ForEach-Object -Process {
$ProviderName = $_
Write-Debug $ProviderName
$xml = $(wevtutil.exe gp /f:xml "$_" 2>$null)
if ($LASTEXITCODE -eq 0 -and $xml) {
$metadata = [xml]$xml
$metadata.Save($(Join-Path -Path $MetadataFolder -ChildPath "$($metadata.provider.guid).xml"));
$metadata.provider.guid | Out-File $(
Join-Path -Path $MetadataSearchByNameFolder -ChildPath "$(_SanitizeFileName $ProviderName).txt")
}
else {
Write-Warning "Invalid metadata for '$ProviderName'"
}
}
function _ResolveKeywords {
param (
[Parameter(Mandatory = $true)]$Metadata,
[Parameter(Mandatory = $true)][ulong]$Keywords
)
if ($Metadata.provider.keywords) {
$Metadata.provider.keywords.keyword | ForEach-Object -Process {
$MaskValue = [ulong]::Parse($_.mask.TrimStart(@('0', 'x', 'X')), [System.Globalization.NumberStyles]::HexNumber)
if ($Keywords -band $MaskValue) {
[PSCustomObject]@{
Name = $_.name
Value = $MaskValue
}
}
}
}
}
# ** EXPORTS **
function Get-EtwProvidersFromWprProfile {
param (
[Parameter(Mandatory = $true)][string]$WprProfilePath
)
if (-not (Test-Path $MetadataFolder)) {
Write-Error "No metadata found - please run Initialize-EtwProvidersMetadata first."
}
function ParseProvider([Parameter(ValueFromPipeline = $true, Mandatory = $true)]$ProviderData) {
begin {}
process {
$MetadataPath = (Join-Path -Path $MetadataFolder -ChildPath "$($ProviderData.Name).xml")
if (-not (Test-Path $MetadataPath)) {
Write-Warning "No metadata found for provider '$($ProviderData.Name)'"
return
}
Write-Debug "Parsing provider '$($ProviderData.Name))'"
$Metadata = [xml](Get-Content $MetadataPath)
[ulong]$Keywords = 0
if ($ProviderData.Keywords) {
$ProviderData.Keywords.Keyword.Value | ForEach-Object -Process {
$Keywords = $Keywords -bor ([ulong]::Parse($_.TrimStart(@('0', 'x', 'X')), [System.Globalization.NumberStyles]::HexNumber)) }
}
else {
$Keywords = [ulong]::MaxValue
}
[ulong]$CaptureOnSaveKeywords = 0
if ($ProviderData.CaptureOnSaveKeywords) {
$ProviderData.CaptureStateOnSave.Keyword.Value | ForEach-Object -Process {
$CaptureOnSaveKeywords = $CaptureOnSaveKeywords -bor ([ulong]::Parse($_.TrimStart(@('0', 'x', 'X')), [System.Globalization.NumberStyles]::HexNumber)) }
}
[PSCustomObject]@{
Id = $ProviderData.Name
Name = $Metadata.provider.name
Keywords = _ResolveKeywords $Metadata $Keywords
CaptureOnSaveKeywords = _ResolveKeywords $Metadata $CaptureOnSaveKeywords
}
}
end {}
}
$xml = [xml](Get-Content $WprProfilePath)
$xml.WindowsPerformanceRecorder.Profiles.EventProvider | ParseProvider
}
function Get-EtwProviderMetadata {
param([Parameter(ValueFromPipeline = $true, Mandatory = $true)]$ProviderName)
$ProviderId = $ProviderName
$Path = $(Join-Path -Path $MetadataSearchByNameFolder -ChildPath "$(_SanitizeFileName $ProviderName).txt")
if (Test-Path $Path) {
$ProviderId = Get-Content $Path
}
$MetadataPath = (Join-Path -Path $MetadataFolder -ChildPath "$ProviderId.xml")
if (-not (Test-Path $MetadataPath)) {
Write-Error "No metadata found for provider '$($ProviderId)'"
}
$Metadata = [xml](Get-Content $MetadataPath)
[PSCustomObject]@{
Id = $ProviderId
Name = $Metadata.provider.name
Keywords = _ResolveKeywords $Metadata $([ulong]::MaxValue)
}
}
================================================
FILE: assets/other/WTComTrace.wprp
================================================
<?xml version="1.0" encoding="utf-8"?>
<WindowsPerformanceRecorder Version="1.0" Author="Sebastian Solnica (https://wtrace.net)" Comments="Profile based on TSS scripts by Microsoft">
<Profiles>
<SystemCollector Id="SystemCollector" Name="NT Kernel Logger">
<BufferSize Value="1024"/>
<Buffers Value="32"/>
</SystemCollector>
<EventCollector Id="EventCollector_MicrosoftWindowsCOMTrace" Name="MicrosoftWindowsCOMTraceCollector">
<BufferSize Value="1024" />
<Buffers Value="32" />
</EventCollector>
<SystemProvider Id="SystemProviderBasic">
<Keywords>
<Keyword Value="ProcessThread" />
<Keyword Value="Loader" />
<Keyword Value="Registry" />
</Keywords>
</SystemProvider>
<EventProvider Id="EventProvider_CombaseTraceLoggingProvider" Name="1AFF6089-E863-4D36-BDFD-3581F07440BE" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_COMSVCS-COMPlus" Name="B46FA1AD-B22D-4362-B072-9F5BA07B046D" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_COMADMIN-COMPlus" Name="A0C4702B-51F7-4ea9-9C74-E39952C694B8" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_COMPlus-Services" Name="53201895-60E8-4fb0-9643-3F80762D658F" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_MicrosoftWindowsComBaseWpp" Name="bda92ae8-9f11-4d49-ba1d-a4c2abca692e" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_MicrosoftWindowsDcomScmWpp" Name="9474a749-a98d-4f52-9f45-5b20247e4f01" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-COM" Name="d4263c98-310c-4d97-ba39-b55354f08584" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-Complus" Name="0f177893-4a9c-4709-b921-f432d67f43d5" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-DistributedCOM" Name="1B562E86-B7AA-4131-BADC-B6F3A001407E" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_CLBCATQ" Name="097d1686-4038-46be-b551-10fda0387165" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-COMRuntime" Name="bf406804-6afa-46e7-8a48-6c357e1d6d61" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-WinRT-Error" Name="A86F8471-C31D-4FBC-A035-665D06047B03" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-WinTypes-Perf" Name="7913ac64-a5cd-40cd-b096-4e8c4028eaab" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-WinRtClassActivation" Name="f0558438-f56a-5987-47da-040ca757ef05" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-RPC" Name="6AD52B32-D609-4BE9-AE07-CE8DAE937E39" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-RPCSS" Name="d8975f88-7ddb-4ed0-91bf-3adf48c48e0c" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-RPC-Events" Name="F4AED7C7-A898-4627-B053-44A7CAA12FCD" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-RPC-Proxy-LBS" Name="272A979B-34B5-48EC-94F5-7225A59C85A0" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-RPC-Proxy" Name="879b2576-39d1-4c0f-80a4-cc086e02548c" NonPagedMemory="true"></EventProvider>
<EventProvider Id="EventProvider_Microsoft-Windows-RPC-LBS" Name="536caa1f-798d-4cdb-a987-05f79a9f457e" NonPagedMemory="true"></EventProvider>
<Profile Id="COMTrace.Verbose.File" Name="COMTrace" Description="COM events trace" LoggingMode="File" DetailLevel="Verbose">
<Collectors>
<SystemCollectorId Value="SystemCollector">
<SystemProviderId Value="SystemProviderBasic"/>
</SystemCollectorId>
<EventCollectorId Value="EventCollector_MicrosoftWindowsCOMTrace">
<EventProviders>
<EventProviderId Value="EventProvider_CombaseTraceLoggingProvider" />
<EventProviderId Value="EventProvider_COMSVCS-COMPlus" />
<EventProviderId Value="EventProvider_COMADMIN-COMPlus" />
<EventProviderId Value="EventProvider_COMPlus-Services" />
<EventProviderId Value="EventProvider_MicrosoftWindowsComBaseWpp" />
<EventProviderId Value="EventProvider_MicrosoftWindowsDcomScmWpp" />
<EventProviderId Value="EventProvider_Microsoft-Windows-COM" />
<EventProviderId Value="EventProvider_Microsoft-Windows-Complus" />
<EventProviderId Value="EventProvider_Microsoft-Windows-DistributedCOM" />
<EventProviderId Value="EventProvider_CLBCATQ" />
<EventProviderId Value="EventProvider_Microsoft-Windows-COMRuntime" />
<EventProviderId Value="EventProvider_Microsoft-Windows-WinRT-Error" />
<EventProviderId Value="EventProvider_Microsoft-Windows-WinTypes-Perf" />
<EventProviderId Value="EventProvider_Microsoft-Windows-WinRtClassActivation" />
<EventProviderId Value="EventProvider_Microsoft-Windows-RPC" />
<EventProviderId Value="EventProvider_Microsoft-Windows-RPCSS" />
<EventProviderId Value="EventProvider_Microsoft-Windows-RPC-Events" />
<EventProviderId Value="EventProvider_Microsoft-Windows-RPC-Proxy-LBS" />
<EventProviderId Value="EventProvider_Microsoft-Windows-RPC-Proxy" />
<EventProviderId Value="EventProvider_Microsoft-Windows-RPC-LBS" />
</EventProviders>
</EventCollectorId>
</Collectors>
<TraceMergeProperties>
<TraceMergeProperty Id="BaseVerboseTraceMergeProperties" Name="BaseTraceMergeProperties">
<DeletePreMergedTraceFiles Value="true" />
<FileCompression Value="false" />
<InjectOnly Value="false" />
<SkipMerge Value="false" />
<CustomEvents>
<CustomEvent Value="ImageId" />
<CustomEvent Value="BuildInfo" />
<CustomEvent Value="VolumeMapping" />
<CustomEvent Value="EventMetadata" />
<CustomEvent Value="PerfTrackMetadata" />
<CustomEvent Value="WinSAT" />
<CustomEvent Value="NetworkInterface" />
</CustomEvents>
</TraceMergeProperty>
</TraceMergeProperties>
</Profile>
</Profiles>
</WindowsPerformanceRecorder>
================================================
FILE: assets/other/winapi-user32.ps1.txt
================================================
$ErrorActionPreference = "Stop"
Add-Type -TypeDefinition @"
using System;
public enum GWL_EXSTYLE : int {
WS_EX_DLGMODALFRAME = 0x00000001,
WS_EX_NOPARENTNOTIFY = 0x00000004,
WS_EX_TOPMOST = 0x00000008,
WS_EX_ACCEPTFILES = 0x00000010,
WS_EX_TRANSPARENT = 0x00000020,
WS_EX_MDICHILD = 0x00000040,
WS_EX_TOOLWINDOW = 0x00000080,
WS_EX_WINDOWEDGE = 0x00000100,
WS_EX_CLIENTEDGE = 0x00000200,
WS_EX_CONTEXTHELP = 0x00000400,
WS_EX_RIGHT = 0x00001000,
WS_EX_LEFT = 0x00000000,
WS_EX_RTLREADING = 0x00002000,
WS_EX_LTRREADING = 0x00000000,
WS_EX_LEFTSCROLLBAR = 0x00004000,
WS_EX_RIGHTSCROLLBAR = 0x00000000,
WS_EX_CONTROLPARENT = 0x00010000,
WS_EX_STATICEDGE = 0x00020000,
WS_EX_APPWINDOW = 0x00040000,
WS_EX_LAYERED = 0x00080000,
WS_EX_NOINHERITLAYOUT = 0x00100000,
WS_EX_NOREDIRECTIONBITMAP = 0x00200000,
WS_EX_LAYOUTRTL = 0x00400000,
WS_EX_COMPOSITED = 0x02000000,
WS_EX_NOACTIVATE = 0x08000000
}
public enum GWL_STYLE : int {
WS_OVERLAPPED = 0x00000000,
WS_POPUP = unchecked((int)0x80000000),
WS_CHILD = 0x40000000,
WS_MINIMIZE = 0x20000000,
WS_VISIBLE = 0x10000000,
WS_DISABLED = 0x08000000,
WS_CLIPSIBLINGS = 0x04000000,
WS_CLIPCHILDREN = 0x02000000,
WS_MAXIMIZE = 0x01000000,
WS_CAPTION = 0x00C00000,
WS_BORDER = 0x00800000,
WS_DLGFRAME = 0x00400000,
WS_VSCROLL = 0x00200000,
WS_HSCROLL = 0x00100000,
WS_SYSMENU = 0x00080000,
WS_THICKFRAME = 0x00040000,
// WS_GROUP = 0x00020000,
// WS_TABSTOP = 0x00010000,
WS_MINIMIZEBOX = 0x00020000,
WS_MAXIMIZEBOX = 0x00010000,
// WS_TILED = WS_OVERLAPPED,
// WS_ICONIC = WS_MINIMIZE,
// WS_SIZEBOX = WS_THICKFRAME
}
public enum SWP : uint {
SWP_NOSIZE = 0x0001,
SWP_NOMOVE = 0x0002,
SWP_NOZORDER = 0x0004,
SWP_NOREDRAW = 0x0008,
SWP_NOACTIVATE = 0x0010,
SWP_FRAMECHANGED = 0x0020,
SWP_SHOWWINDOW = 0x0040,
SWP_HIDEWINDOW = 0x0080,
SWP_NOCOPYBITS = 0x0100,
SWP_NOOWNERZORDER = 0x0200,
SWP_NOSENDCHANGING = 0x0400,
// SWP_DRAWFRAME = SWP_FRAMECHANGED,
// SWP_NOREPOSITION = SWP_NOOWNERZORDER,
SWP_DEFERERASE = 0x2000,
SWP_ASYNCWINDOWPOS = 0x4000
}
"@
================================================
FILE: assets/other/windbg-install.ps1.txt
================================================
# script created by @Izybkr (https://github.com/microsoftfeedback/WinDbg-Feedback/issues/19#issuecomment-1513926394) with my minor updates to make it work with latest WinDbg releases):
param(
$OutDir = ".",
[ValidateSet("x64", "x86", "arm64")]
$Arch = "x64"
)
if (!(Test-Path $OutDir)) {
$null = mkdir $OutDir
}
$ErrorActionPreference = "Stop"
if ($PSVersionTable.PSVersion.Major -le 5) {
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# This is a workaround to get better performance on older versions of PowerShell
$ProgressPreference = 'SilentlyContinue'
}
# Download the appinstaller to find the current uri for the msixbundle
Invoke-WebRequest https://aka.ms/windbg/download -OutFile $OutDir\windbg.appinstaller
# Download the msixbundle
$msixBundleUri = ([xml](Get-Content $OutDir\windbg.appinstaller)).AppInstaller.MainBundle.Uri
# Download the msixbundle (but name as zip for older versions of Expand-Archive
Invoke-WebRequest $msixBundleUri -OutFile $OutDir\windbg.zip
# Extract the 3 msix files (plus other files)
Expand-Archive -DestinationPath $OutDir\UnzippedBundle $OutDir\windbg.zip
# Expand the build you want - also renaming the msix to zip for Windows PowerShell
$fileName = switch ($Arch) {
"x64" { "windbg_win-x64" }
"x86" { "windbg_win-x86" }
"arm64" { "windbg_win-arm64" }
}
# Rename msix (for older versions of Expand-Archive) and extract the debugger
Rename-Item "$OutDir\UnzippedBundle\$fileName.msix" "$fileName.zip"
Expand-Archive -DestinationPath "$OutDir\windbg" "$OutDir\UnzippedBundle\$fileName.zip"
Remove-Item -Recurse -Force "$OutDir\UnzippedBundle"
Remove-Item -Force "$OutDir\windbg.appinstaller"
Remove-Item -Force "$OutDir\windbg.zip"
# Now you can run:
& $OutDir\windbg\DbgX.Shell.exe
================================================
FILE: browserconfig.xml
================================================
<?xml version="1.0" encoding="utf-8"?>
<browserconfig>
<msapplication>
<tile>
<square150x150logo src="/mstile-150x150.png"/>
<TileColor>#da532c</TileColor>
</tile>
</msapplication>
</browserconfig>
================================================
FILE: guides/com-troubleshooting.md
================================================
---
layout: page
title: COM troubleshooting
date: 2023-04-07 08:00:00 +0200
redirect_from:
- /articles/com-troubleshooting/
- /articles/com-troubleshooting
---
{% raw %}
**Table of contents:**
<!-- MarkdownTOC -->
- [Quick introduction to COM](#quick-introduction-to-com)
- [COM metadata](#com-metadata)
- [Troubleshooting COM in WinDbg](#troubleshooting-com-in-windbg)
- [Monitoring COM objects in a process](#monitoring-com-objects-in-a-process)
- [Tracing COM methods](#tracing-com-methods)
- [Stopping the COM monitor](#stopping-the-com-monitor)
- [Observing COM interactions outside WinDbg](#observing-com-interactions-outside-windbg)
- [Windows Performance Recorder \(wpr.exe\)](#windows-performance-recorder-wprexe)
- [Process Monitor](#process-monitor)
- [wtrace](#wtrace)
- [Troubleshooting .NET COM interop](#troubleshooting-net-com-interop)
- [Links](#links)
<!-- /MarkdownTOC -->
Quick introduction to COM
-------------------------
In COM, everything is about interfaces. In old times, when various compiler vendors were fighting over whose "standard" was better, the only reliable way to call C++ class methods contained in third-party libraries was to use virtual tables. As its name suggests virtual table is a table, to be precise, a table of addresses (pointers). The "virtual" adjective relates to the fact that our table's addresses point to virtual methods. If you're familiar with object programming (you plan to debug COM, so you should!), you probably thought of inheritance and abstract classes. And that's correct! The abstract class is how we implement interfaces in C++ (to be more precise [an abstract class with pure virtual methods](https://en.cppreference.com/w/cpp/language/abstract_class)). Now, COM is all about passing pointers to those various virtual tables which happen to have GUID identifiers. The most important interface (parent of all interfaces) is `IUnknown`. Every COM interface must inherit from this interface. Why? For two reasons: to manage the object lifetime and to access all the other interfaces that our object may implement (or, in other words, to find all virtual tables our object is aware of). As this interface is so important, let's have a quick look at its definition:
```cpp
struct __declspec(uuid("00000000-0000-0000-C000-000000000046"))) IUnknown
{
public:
virtual HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, void **ppvObject) = 0;
virtual ULONG STDMETHODCALLTYPE AddRef( void) = 0;
virtual ULONG STDMETHODCALLTYPE Release( void) = 0;
};
```
Guess which methods are responsible for lifetime management and which are for interface querying. OK, so we know the declaration, but to debug COM, we need to understand how COM objects are laid out in the memory. Let's have a look at a sample Probe class (the snippet comes from [my Protoss COM example repository](https://github.com/lowleveldesign/protoss-com-example)):
```cpp
struct __declspec(uuid("59644217-3e52-4202-ba49-f473590cc61a")) IGameObject : public IUnknown
{
public:
virtual HRESULT STDMETHODCALLTYPE get_Name(BSTR* name) = 0;
virtual HRESULT STDMETHODCALLTYPE get_Minerals(LONG* minerals) = 0;
virtual HRESULT STDMETHODCALLTYPE get_BuildTime(LONG* buildtime) = 0;
};
struct __declspec(uuid("246A22D5-CF02-44B2-BF09-AAB95A34E0CF")) IProbe : public IUnknown
{
public:
virtual HRESULT STDMETHODCALLTYPE ConstructBuilding(BSTR building_name, IUnknown * *ppUnk) = 0;
};
class __declspec(uuid("EFF8970E-C50F-45E0-9284-291CE5A6F771")) Probe final : public IProbe, public IGameObject
{
ULONG ref_count;
/* ... implementation .... */
}
```
If we instantiate (more on that later) the Probe class, its layout in the memory will look as follows:
```
0:000> dps 0xfb2f58 L4
00fb2f58 72367744 protoss!Probe::`vftable'
00fb2f5c 7236775c protoss!Probe::`vftable'
00fb2f60 00000001
00fb2f64 fdfdfdfd
0:000> dps 72367744 L4 * IProbe interface
72367744 72341bb3 protoss!ILT+2990(?QueryInterfaceProbeUAGJABU_GUIDPAPAXZ)
72367748 72341ba9 protoss!ILT+2980(?AddRefProbeUAGKXZ)
7236774c 723411ae protoss!ILT+425(?ReleaseProbeUAGKXZ)
72367750 723414d3 protoss!ILT+1230(?ConstructBuildingProbeUAGJPA_WPAPAUIUnknownZ)
0:000> dps 7236775c L6 * IGameUnit interface
7236775c 72341e3d protoss!ILT+3640(?QueryInterfaceProbeW3AGJABU_GUIDPAPAXZ)
72367760 723416fe protoss!ILT+1785(?AddRefProbeW3AGKXZ)
72367764 72341096 protoss!ILT+145(?ReleaseProbeW3AGKXZ)
72367768 723415f0 protoss!ILT+1515(?get_NameProbeUAGJPAPA_WZ)
7236776c 723419d8 protoss!ILT+2515(?get_MineralsProbeUAGJPAJZ)
72367770 72341e1a protoss!ILT+3605(?get_BuildTimeProbeUAGJPAJZ)
```
Notice the pointers at the beginning of the object memory. As you can see in the snippet, those pointers reference arrays of function pointers or, as you remember, virtual tables. Each virtual table represents a COM interface, like `IProbe` or `IGameObject` in our case.
Let's now briefly discuss the creation of COM objects. We usually start by calling one of the well-known Co-functions to create a COM object. Often, it's either `CoCreateInstance` or `CoGetClassObject`. Those functions perform actions defined in the COM registration (either in a manifest file or in the registry). In the most common (and most straightforward scenario), they load a dll and run the exported `DllGetClassObject` function:
```cpp
HRESULT DllGetClassObject([in] REFCLSID rclsid, [in] REFIID riid, [out] LPVOID *ppv);
```
On a successful return, the `*ppv` value should point to an address of the virtual table representing a COM interface with the IID equal to `riid`. And this address will be a part of memory belonging to a COM object of the type identified by the `rclsid`.
People often say that COM is complicated. As you just saw, COM fundamentals are clear and straightforward. However, its various implementations might cause a headache. For example, there are myriads of methods in OLE and ActiveX interfaces created to make it possible to drag/drop things between windows, use the clipboard, or embed one control in another. Remember, though, that all those crazy interfaces still need to implement `IUnknown`. And that's the advantage we can take as troubleshooters. It's easy to track new instance creations, interface queries, and interface method calls (often even with their names). That may give us enough insights to debug a problem successfully.
### COM metadata
COM metadata, saved in type libraries, provides definitions of COM classes and COM interfaces. Thanks to it, we can decode method names and their argument values without debugging symbols. The tool we usually use to view the type libraries installed in the system is [OleView](https://learn.microsoft.com/en-us/windows/win32/com/ole-com-object-viewer), part of the Windows SDK. OleView has some open-source alternatives, such as [.NET OLE/COM viewer](https://github.com/tyranid/oleviewdotnet) or [OleWoo](https://github.com/leibnitz27/olewoo). [Comon](https://github.com/lowleveldesign/comon) also provides the **!cometa** command, which allows you to use COM metadata without leaving WinDbg. Before the debugging session, it is worth taking a moment to build the cometa database with the **!cometa index** command. The database resides in a temporary folder. It's an SQLite database, so you may copy it between machines. Other comon commands will use the cometa database to resolve class and interface IDs to meaningful names.
You may also do some basic queries against the database with the **!cometa showc** and **!cometa showi** commands, for example:
```
0:000> !cometa showi {59644217-3E52-4202-BA49-F473590CC61A}
Found: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject)
Methods:
- [0] HRESULT QueryInterface(void* this, GUID* riid, void** ppvObject)
- [1] ULONG AddRef(void* this)
- [2] ULONG Release(void* this)
- [3] HRESULT get_Name(void* this, BSTR* Name)
- [4] HRESULT get_Minerals(void* this, long* Minerals)
- [5] HRESULT get_BuildTime(void* this, long* BuildTime)
Registered VTables for IID:
- Module: protoss, CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Probe), VTable offset: 0x3775c
- Module: protoss, CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Nexus), VTable offset: 0x37710
```
Troubleshooting COM in WinDbg
-----------------------------
### Monitoring COM objects in a process
There are various ways in which COM objects can be created. When a given function creates a COM object, you will see a `void **` as one of its arguments. After a successful call, this pointer will point to a new COM object. Let's check how we can trace such a creation. We will use breakpoints to monitor calls to the `CoCreateInstance(REFCLSID rclsid, LPUNKNOWN pUnkOuter, DWORD dwClsContext, REFIID riid, LPVOID *ppv)` function. We are interested in the class (`rclsid`) and interface (`riid`) values, and the address of the created COM object (`*ppv`). When debugging a 64-bit process, our breakpoint command might look as follows:
```
bp combase!CoCreateInstance ".echo ==== combase!CoCreateInstance ====; dps @rsp L8; dx *(combase!GUID*)@rcx; dx *(combase!GUID*)@r9; .printf /D \"==> obj addr: %p\", poi(@rsp+28);.echo; bp /1 @$ra; g"
```
The `bp /1 @$ra` part creates a one-time breakpoint at a function return address. This second breakpoint will stop the process execution and allow us to examine the results of the function call. At this time, the `rax` register will show the return code (should be `0` for a successful call), and the created COM object (and also the interface virtual) will be at the previously printed object address. For the sake of completeness, let me show you the 32-bit version of this breakpoint:
```
bp combase!CoCreateInstance ".echo ==== combase!CoCreateInstance ====; dps @esp L8; dx **(combase!GUID **)(@esp + 4); dx **(combase!GUID **)(@esp + 0x10); .printf /D \"==> obj addr: %p\", poi(@esp+14);.echo; bp /1 @$ra; g"
```
Creating such breakpoints for various COM functions might be a mundane task, especially when we consider that our only point in doing so is to save the addresses of the virtual tables. **Fortunately, [comon](https://github.com/lowleveldesign/comon) might be of help here**. In-process COM creation usually ends in a call to the `DllGetClassObject` function exported by the DLL implementing a given COM object. After **attaching to a process** (**!comon attach**), comon creates breakpoints on all such functions and checks the results of their executions. It also breaks when a process calls `CoRegisterClassObject`, a function called by out-of-process COM servers to register the COM objects they host.
After you attach comon to a debugged process, you should see various log messages showing COM object creations, for example:
```
0:000> !comon attach
COM monitor enabled for the current process.
0:000> g
...
[comon] 0:000 [protoss!DllGetClassObject] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {00000001-0000-0000-C000-000000000046} (IClassFactory) -> SUCCESS (0x0)
[comon] 0:000 [IClassFactory::CreateInstance] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {246A22D5-CF02-44B2-BF09-AAB95A34E0CF} (IProbe) -> SUCCESS (0x0)
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Protoss Probe), IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject) -> SUCCESS (0x0)
[comon] 0:000 [protoss!DllGetClassObject] CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Protoss Nexus), IID: {00000001-0000-0000-C000-000000000046} (IClassFactory) -> SUCCESS (0x0)
[comon] 0:000 [IClassFactory::CreateInstance] CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Protoss Nexus), IID: {C5F45CBC-4439-418C-A9F9-05AC67525E43} (INexus) -> SUCCESS (0x0)
[comon] 0:000 [IUnknown::QueryInterface] CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Protoss Nexus), IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject) -> SUCCESS (0x0)
...
```
The `QueryInterface` calls will show up only for the first time; it won't be reported if we have the virtual table for a given interface already registered in the cometa database. To check the COM objects registered in a given session, run the **!comon status** command, for example:
```
0:000> !comon status
COM monitor is RUNNING
COM types recorded for the current process:
CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Nexus)
IID: {C5F45CBC-4439-418C-A9F9-05AC67525E43} (INexus), address: 0x723676f8
IID: {00000001-0000-0000-C000-000000000046} (N/A), address: 0x7236694c
IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject), address: 0x72367710
CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Probe)
IID: {00000001-0000-0000-C000-000000000046} (N/A), address: 0x72366968
IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject), address: 0x7236775c
IID: {246A22D5-CF02-44B2-BF09-AAB95A34E0CF} (IProbe), address: 0x72367744
```
The `cometa` queries show now also return information about the registered virtual tables:
```
0:000> !cometa showc {F5353C58-CFD9-4204-8D92-D274C7578B53}
Found: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Nexus)
Registered VTables for CLSID:
- module: protoss, IID: {00000001-0000-0000-C000-000000000046} (N/A), VTable offset: 0x3694c
- module: protoss, IID: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject), VTable offset: 0x37710
- module: protoss, IID: {C5F45CBC-4439-418C-A9F9-05AC67525E43} (INexus), VTable offset: 0x376f8
```
### Tracing COM methods
When we know the interface virtual table address, nothing can stop us from creating breakpoints on interface methods :) I will first show you how to do that manually and later present how [comon](https://github.com/lowleveldesign/comon) may help.
The first step is to find the offset of our method in the interface definition. Let's stick to the Protoss COM example and let's create a breakpoint on the `get_Minerals` method/property from the `IGameObject` interface:
```
0:000> !cometa showi {59644217-3E52-4202-BA49-F473590CC61A}
Found: {59644217-3E52-4202-BA49-F473590CC61A} (IGameObject)
Methods:
- [0] HRESULT QueryInterface(void* this, GUID* riid, void** ppvObject)
- [1] ULONG AddRef(void* this)
- [2] ULONG Release(void* this)
- [3] HRESULT get_Name(void* this, BSTR* Name)
- [4] HRESULT get_Minerals(void* this, long* Minerals)
- [5] HRESULT get_BuildTime(void* this, long* BuildTime)
Registered VTables for IID:
- Module: protoss, CLSID: {EFF8970E-C50F-45E0-9284-291CE5A6F771} (Probe), VTable offset: 0x3775c
- Module: protoss, CLSID: {F5353C58-CFD9-4204-8D92-D274C7578B53} (Nexus), VTable offset: 0x37710
```
We can see that its ordinal number is four, and two virtual tables are registered for our interface (two classes implementing it). Let's focus on the `Probe` class. To set a breakpoint method, we can use the `bp` command:
```
bp poi(protoss + 0x3775c + 4 * $ptrsize)
```
Similarly, if we would like to set breakpoints on all the `IGameObject` methods, we might use a loop:
```
.for (r $t0 = 0; @$t0 < 6; r $t0 = @$t0 + 1) { bp poi(protoss + 0x3775c + @$t0 * @$ptrsize) }
```
Instead of setting breakpoints manually, you may use the **!cobp** command from the comon extension. It also creates a breakpoint (you will see it if you run the bl command), but on hit, comon will decode the method parameters (for the supported types). It will also automatically create a one-time breakpoint on the method return address, displaying the return code and method out parameter values. The optional parameter lets you decide if you'd like to stop when cobreakpoint is hit. An example output might look as follows:
```
0:000> !cobp --always {EFF8970E-C50F-45E0-9284-291CE5A6F771} {59644217-3E52-4202-BA49-F473590CC61A} get_Name
[comon] Breakpoint 18 (address 0x723415f0) created / updated
0:000> g
[comon breakpoint] IGameObject::get_Name (iid: {59644217-3E52-4202-BA49-F473590CC61A}, clsid: {EFF8970E-C50F-45E0-9284-291CE5A6F771})
Parameters:
- this: 0xfb2f5c (void*)
- Name: 0x81fc1c (BSTR*) [out]
0:000> dps 0081fc1c L1
0081fc1c 00000000
0:000> g
[comon breakpoint] IGameObject::get_Name (iid: {59644217-3E52-4202-BA49-F473590CC61A}, clsid: {EFF8970E-C50F-45E0-9284-291CE5A6F771}) return
Result: 0x0 (HRESULT)
Out parameters:
- Name: 0x81fc1c (BSTR*)
0:000> du 00f9c6ac
00f9c6ac "Probe"
```
If comon can't decode a given parameter, you may use the **dx** command with combase.dll symbols (one of the rare Microsoft DLLs that comes with private symbols), for example: `dx -r2 (combase!DISPPARAMS *)(*(void **)(@esp+0x18))` or `dx -r1 ((combase!tagVARIANT[3])0x31ec1f0)`.
### Stopping the COM monitor
Run the **!comon detach** command to stop the COM monitor. This command will remove all the comon breakpoints and debugging session data, but you can still examine COM metadata with the cometa command.
Observing COM interactions outside WinDbg
-----------------------------------------
Sometimes we only need basic information about COM interactions, such as which objects are used and how they are launched. While WinDbg can be overkill for such scenarios, there are several simpler tools we can use to collect this additional information.
### Windows Performance Recorder (wpr.exe)
Let's begin with wpr.exe, a powerful tool that's likely already installed on your system. WPR requires profile files to configure tracing sessions. For basic COM event collection, you can use [the ComTrace.wprp profile](https://raw.githubusercontent.com/microsoft/winget-cli/refs/heads/master/tools/COMTrace/ComTrace.wprp) from [the winget-cli repository](https://github.com/microsoft/winget-cli). I've also created an enhanced profile, adding providers found in the [TSS scripts](https://learn.microsoft.com/en-us/troubleshoot/windows-client/windows-tss/introduction-to-troubleshootingscript-toolset-tss), which you can download **[here](/assets/other/WTComTrace.wprp)**. You can use those profiles either solely or in combination with other profiles, as shown in the examples below.
```shell
# Collect only COM events
wpr.exe -start .\WTComTrace.wprp -filemode
# Run COM apps ...
# Stop the trace when done
wpr -stop C:\temp\comtrace.etl
# Collect COM events with CPU sampling
wpr.exe -start CPU -start .\WTComTrace.wprp -filemode
# Run COM apps ...
# Stop the trace when done
wpr -stop C:\temp\comtrace.etl
```
Some providers are the [legacy WPP providers](https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/wpp-software-tracing), which require TMF files to read the collected events. Fortunately, the PDB file for compbase.dll contains the required TMF data and we can decode those events. To view the collected data, open the ETL file in **[Windows Performance Analyzer (WPA)](https://learn.microsoft.com/en-us/windows-hardware/test/wpt/windows-performance-analyzer)**. Remember to load symbols first (check [the Windows configuration guide](guides/configuring-windows-for-effective-troubleshooting/#configuring-debug-symbols) how to configure symbols globally in the system), then navigate to the **Generic Events** category and open the **WPP Trace** view.
### Process Monitor
In **[Process Monitor](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon)**, we can include Registry and Process events and events where Path contains `\CLSID\` or `\AppID` strings or ends with `.dll`, as in the image below:
The collected events should tell us which COM objects the application initiated and in which way. For example, if procmon shows a DLL path read from the `InprocServer32` and then we see this dll loaded, we may assume that the application created a given COM object (the event call stack may be an additional proof). If the COM server runs in a standalone process or a remote machine, other keys will be queried. We may then check the Process Tree or Network events for more details. [COM registry keys official documentation](https://learn.microsoft.com/en-us/windows/win32/com/com-registry-keys) is thorough, so please consult it to learn more.
### wtrace
In **[wtrace](https://github.com/lowleveldesign/wtrace)**, we need to pick the proper handlers and define filters. An example command line might look as follows:
```shell
wtrace --handlers registry,process,rpc -f 'path ~ \CLSID\' -f 'path ~ \AppID\' -f 'path ~ rpc' -f 'pname = ProtossComClient'
```
As you can see, wtrace may additionally show information about RPC (Remote Procedure Call) events.
Troubleshooting .NET COM interop
--------------------------------
A native COM object must be wrapped into a Runtime Callable Wrapper (RCW) to be accessible to managed code. RCW binds a managed object (for example, `System.__Com`) and a native COM class instance. COM Callable Wrappers (CCW) work in the opposite direction - thanks to them, we may expose .NET objects to the COM world. Interestingly, the object interop usage is saved in the object's SyncBlock. Therefore, it should not come as a surprise that the **!syncblk** command from [the SOS extension](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/sos-debugging-extension) presents information about RCWs and CCWs:
```
0:011> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
-----------------------------
Total 5
CCW 1
RCW 0
ComClassFactory 0
Free 3
```
When we add the **-all** parameter, **!syncblk** will list information about the created SyncBlocks with their corresponding objects, for example:
```
0:007> !syncblk -all
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
1 07FF8F54 0 0 00000000 none 030deb48 System.__ComObject
2 07FF8F20 0 0 00000000 none 030deb3c EventTesting
3 00000000 0 0 00000000 none 0 Free
4 00000000 0 0 00000000 none 0 Free
5 00000000 0 0 00000000 none 0 Free
-----------------------------
Total 5
CCW 1
RCW 0
ComClassFactory 0
Free 3
```
Now, we can dump information about managed objects using the **!dumpobj** command, for example:
```
0:006> !dumpobj 030deb3c
Name: EventTesting
MethodTable: 08301668
EEClass: 082f7110
CCW: 0833ffe0
Tracked Type: false
Size: 12(0xc) bytes
File: c:\repos\testing-com-events\bin\NETServer.dll
Fields:
MT Field Offset Type VT Attr Value Name
0830db50 4000003 4 ...ng+OnEventHandler 0 instance 00000000 onEvent```
```
The good news is that the **!dumpobj** command also checks if a given object has a SyncBlock assigned and dumps information from it. In this case, it's the address of CCW. We may get more details about it by using the **!dumpccw** command:
```
0:011> !dumpccw 08060000
Managed object: 02e6cf88
Outer IUnknown: 00000000
Ref count: 0
Flags:
RefCounted Handle: 00D714F8 (WEAK)
COM interface pointers:
IP MT Type
08060010 080315b0 Server.Contract.IEventTesting
```
Notice here that there is only one interface implemented by the managed object and the CCW is no longer in use by the native code (Ref count equals 0). Below is an example of a CCW representing a Windows Forms ActiveX control which is still alive and implements more interfaces:
```
0:014> !dumpccw 0a23fde0
Managed object: 04ee6984
Outer IUnknown: 00000000
Ref count: 7
Flags:
RefCounted Handle: 04C716D8 (STRONG)
COM interface pointers:
IP MT Type
0A23FDF8 09fbbb04 Interop+Ole32+IOleControl
0A23FDC8 09fbbc4c Interop+Ole32+IOleObject
0A23FDCC 09fbbd34 Interop+Ole32+IOleInPlaceObject
0A23FDD0 09fbbde4 Interop+Ole32+IOleInPlaceActiveObject
0A23FDA8 09fbbfa0 Interop+Ole32+IViewObject2
0A23FDB0 09fbc09c Interop+Ole32+IPersistStreamInit
0A23FD4C 09f6485c BullsEyeControlLib.IBullsEye
```
If you would like to dump information about all objects associated with SyncBlocks, you may use the following WinDbg script:
```
.foreach /pS 7 /ps 7 (addr { !syncblk -all }) { !do addr }
```
And to extract only the RCW or CCW addresses, we could use the **!grep** command from the [awesome Andrew Richard's PDE extension](https://onedrive.live.com/?authkey=%21AJeSzeiu8SQ7T4w&id=DAE128BD454CF957%217152&cid=DAE128BD454CF957):
```
0:014> .load PDE.dll
0:014> !grep RCW: .foreach /pS 7 /ps 7 (addr { !syncblk -all }) { !do addr }
RCW: 08086d30
0:014> !grep CCW: .foreach /pS 7 /ps 7 (addr { !syncblk -all }) { !do addr }
CCW: 08060000
```
To keep COM objects alive in the managed memory, .NET Runtime creates handles for them. Those are either strong or ref-counted handles and we may list them with the **!gchandles** command, for example:
```
0:011> !gchandles -type refcounted
Handle Type Object Size Data Type
00D714F8 RefCounted 02e6cf88 12 0 EventTesting
Statistics:
MT Count TotalSize Class Name
08031668 1 12 EventTesting
Total 1 objects
0:014> !gchandles -type strong
Handle Type Object Size Data Type
04C711B4 Strong 030deb48 12 System.__ComObject
...
Statistics:
MT Count TotalSize Class Name
04ebbf00 1 12 System.__ComObject
...
Total 19 objects
```
Of course, in those lists we will find the objects we already saw in the **!syncblk** output, so it's just another way to find them. It may be useful when tracking, for example, GC leaks.
Finally, to find who is keeping our managed object alive, we could use the **!gcroot** command. And it's quite easy to find the GC roots for a particular type with the following script:
```
.foreach (addr { !DumpHeap -short -type System.__ComObject }) { !gcroot addr }
```
Links
-----
- ["Essential COM"](https://archive.org/details/essentialcom00boxd) by Don Box
- ["Inside OLE"](https://github.com/kraigb/InsideOLE) by Kraig Brockschmidt (Kraig published the whole book with source code on GitHub!)
- ["Inside COM+ Base Services"](https://thrysoee.dk/InsideCOM+/) by Guy Eddon and Henry Eddon
- ["COM and .NET interoperability"](https://link.springer.com/book/10.1007/978-1-4302-0824-2) and [source code](https://github.com/Apress/com-.net-interoperability) by Andrew Troelsen
- [".NET and COM: The Complete Interoperability Guide"](https://books.google.pl/books/about/NET_and_COM.html?id=x2OIPSyFLBcC) by Adam Nathan
- [COM+ revisited](https://lowleveldesign.wordpress.com/2022/01/17/com-revisited/) by me :)
- [Calling Local Windows RPC Servers from .NET](https://googleprojectzero.blogspot.com/2019/12/calling-local-windows-rpc-servers-from.html) by James Forshaw
{% endraw %}
================================================
FILE: guides/configuring-linux-for-effective-troubleshooting.md
================================================
---
layout: page
title: Configuring Linux for effective troubleshooting
date: 2025-12-26 08:00:00 +0200
---
**Table of contents:**
<!-- MarkdownTOC -->
- [Configuring debug symbols](#configuring-debug-symbols)
<!-- /MarkdownTOC -->
Configuring debug symbols
-------------------------
These days many debugging tools can fetch debug symbols from debuginfod servers. The [official project page](https://sourceware.org/elfutils/Debuginfod.html) lists the URLs you should use for each supported distribution. For example, in my Arch Linux, the `DEBUGINFOD_URLS` environment variable is set to `https://debuginfod.archlinux.org` by the `/etc/profile.d/debuginfod.sh` script (a part of the libelf package).
If you want this variable to be preserved when running commands with sudo, you can add a rule such as the following to a file in `/etc/sudoers.d/` (e.g., `/etc/sudoers.d/debuginfod`):
```
Defaults env_keep += "DEBUGINFOD_URLS"
```
================================================
FILE: guides/configuring-windows-for-effective-troubleshooting.md
================================================
---
layout: page
title: Configuring Windows for effective troubleshooting
date: 2023-10-11 08:00:00 +0200
---
**Table of contents:**
<!-- MarkdownTOC -->
- [Configuring debug symbols](#configuring-debug-symbols)
- [Replacing Task Manager with System Informer](#replacing-task-manager-with-system-informer)
- [Installing and configuring Sysinternals Suite](#installing-and-configuring-sysinternals-suite)
- [Configuring post-mortem debugging](#configuring-post-mortem-debugging)
<!-- /MarkdownTOC -->
## Configuring debug symbols
Staring at raw hex numbers is not very helpful for troubleshooting. Therefore, it's essential to take the time to properly configure debug symbols on our system. One effective method is to set the **\_NT\_SYMBOL\_PATH** environment variable. Most troubleshooting tools read its value and utilize the specified symbol stores. I usually configure it to point only to the official Microsoft symbol server, resulting in the following value for the \_NT\_SYMBOL\_PATH variable on my system: `SRV*C:\symbols\dbg*https://msdl.microsoft.com/download/symbols`. Here, `C:\symbols` serves as a cache folder for storing downloaded symbols. I also use `C:\symbols\dbg` if I need to index PDB files for my applications. For further information about the \_NT\_SYMBOL\_PATH variable, refer to [the official documentation](https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/symbol-path).
The symbol path variable is one essential component required for successful symbol resolution. Another critical aspect is the version of **dbghelp.dll** that can work with symbol servers. Unfortunately, the version preinstalled with Windows lacks this feature. To overcome this issue, you can install the **Debugging Tools for Windows** from the [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/). Make sure to install both the x86 and x64 versions to enable debugging of both 32- and 64-bit applications. Once installed, certain tools (e.g., Symbol Informer) will automatically select the appropriate dbghelp.dll version, while others will require some configuration, as we'll explore in later sections.
## Replacing Task Manager with System Informer
My long time favorite tool to observe system and processes running on it, is [System Informer](https://www.systeminformer.com/), formerly known as Process Hacker. It has so many great features that deserves a guide on its own. The process tree, which shows the process creation and termination events, is much more readable than the flat process list in Task Manager or Resource Monitor. Moreover, System Informer lets you manage services and drivers, and view live network connections. Therefore, I highly recommend to open the Options dialog and replace Task Manager with it. System Informer does not have an option to set the dbghelp.dll path in its settings, but it will detect it if you have Debugging Tools for Windows installed. So please install them to have Windows stacks correctly resolved.
If you have reasons not to use System Informer, you can try [Process Explorer](https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer). It does not have as many functionalities as System Informer, but it is still a powerful system monitor.
## Installing and configuring Sysinternals Suite
[Sysinternals tools](https://learn.microsoft.com/en-us/sysinternals/) help me diagnose and fix various issues on Windows systems. Most often I use [Process Monitor](https://learn.microsoft.com/en-us/sysinternals/downloads/procmon) to capture and analyze system events, and sometimes that's the only tool I need to solve the problem! Other Sysinternals tools that I frequently use are [DebugView](https://learn.microsoft.com/en-us/sysinternals/downloads/debugview), [ProcDump](https://learn.microsoft.com/en-us/sysinternals/downloads/procdump), and [LiveKd](https://learn.microsoft.com/en-us/sysinternals/downloads/livekd). You can get the entire suite or individual tools from the [SysInternals website](https://learn.microsoft.com/en-us/sysinternals/downloads/) or from [live.sysinternals.com](https://live.sysinternals.com). However, these methods require manual updates when new versions are available. A more convenient way to keep the tools up to date is to install them from [Microsoft Store](https://www.microsoft.com/store/apps/9p7knl5rwt25).
To get the most out of Process Monitor and Process Explorer, you need to set up symbol resolution correctly. The default settings do not use the Microsoft symbol store, so you need to adjust them in the options or import the registry keys shown below (after installing Debugging Tools for Windows):
```
[HKEY_CURRENT_USER\Software\Sysinternals\Process Explorer]
"DbgHelpPath"="C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\dbghelp.dll"
"SymbolPath"="SRV*C:\\symbols\\dbg*http://msdl.microsoft.com/download/symbols"
[HKEY_CURRENT_USER\Software\Sysinternals\Process Monitor]
"DbgHelpPath"="C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\dbghelp.dll"
"SymbolPath"="SRV*C:\\symbols\\dbg*http://msdl.microsoft.com/download/symbols"
```
## Configuring post-mortem debugging
We all experience application failures from time to time. When it happens, Windows collectes some data about a crash and saves it to the event log. It usually lacks details required to fully understand the root cause of an issue. Fortunately, we have options to replace this scarse report with, for example, a memory dump. One way to accomplish that is by configuring **Windows Error Reporting** . The commands below will enable minidump collection to a C:\Dumps folder on a process failure:
```shell
reg.exe add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps" /v DumpType /t REG_DWORD /d 1 /f
reg.exe add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps" /v DumpFolder /t REG_EXPAND_SZ /d C:\dumps /f
```
The available settings are listed and explained in the [WER documentation](https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps). Note, that by creating a subkey with an application name (for example, `HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\test.exe`), you may customize WER settings per individual applications.
[ProcDump](https://learn.microsoft.com/en-us/sysinternals/downloads/procdump) is an alternative to WER. You could install it as an [automatic debugger](https://learn.microsoft.com/en-us/windows/win32/debug/configuring-automatic-debugging), which Windows will run whenever a critical error occurs in an application. Example install command (-u to uninstall):
```shell
procdump -i C:\Dumps
```
These dumps can take up a lot of disk space over time, so you should either delete the old files periodically, or set up a task scheduler job that does it for you.
================================================
FILE: guides/diagnosing-dotnet-apps.md
================================================
---
layout: page
title: Diagnosing .NET applications
date: 2024-01-01 08:00:00 +0200
---
{% raw %}
:point_right: I also authored the **[.NET Diagnostics Expert](https://diagnosticsexpert.com/?utm_source=debugrecipes&utm_medium=banner&utm_campaign=general) course**, available at Dotnetos :hot_pepper: Academy. Apart from the theory, it contains lots of demos and troubleshooting guidelines. Check it out if you're interested in learning .NET troubleshooting. :point_left:
**Table of contents:**
<!-- MarkdownTOC -->
- [General .NET debugging tips](#general-net-debugging-tips)
- [Loading the SOS extension into WinDbg](#loading-the-sos-extension-into-windbg)
- [Manually loading symbol files for .NET Core](#manually-loading-symbol-files-for-net-core)
- [Disabling JIT optimization](#disabling-jit-optimization)
- [Decoding managed stacks in Sysinternals](#decoding-managed-stacks-in-sysinternals)
- [Check runtime version](#check-runtime-version)
- [Debugging/tracing a containerized .NET application \(Docker\)](#debuggingtracing-a-containerized-net-application-docker)
- [Diagnosing exceptions or erroneous behavior](#diagnosing-exceptions-or-erroneous-behavior)
- [Using Time Travel Debugging \(TTD\)](#using-time-travel-debugging-ttd)
- [Collecting a memory dump](#collecting-a-memory-dump)
- [Analysing exception information](#analysing-exception-information)
- [Diagnosing hangs](#diagnosing-hangs)
- [Listing threads call stacks](#listing-threads-call-stacks)
- [Finding locks in managed code](#finding-locks-in-managed-code)
- [Diagnosing waits or high CPU usage](#diagnosing-waits-or-high-cpu-usage)
- [Diagnosing managed memory leaks](#diagnosing-managed-memory-leaks)
- [Collecting memory snapshots](#collecting-memory-snapshots)
- [Analyzing collected snapshots](#analyzing-collected-snapshots)
- [Diagnosing issues with assembly loading](#diagnosing-issues-with-assembly-loading)
- [Troubleshooting loading with EventPipes/ETW \(.NET\)](#troubleshooting-loading-with-eventpipesetw-net)
- [Troubleshooting loading using ETW \(.NET Framework\)](#troubleshooting-loading-using-etw-net-framework)
- [Troubleshooting loading using Fusion log \(.NET Framework\)](#troubleshooting-loading-using-fusion-log-net-framework)
- [GAC \(.NET Framework\)](#gac-net-framework)
- [Find assembly in cache](#find-assembly-in-cache)
- [Uninstall assembly from cache](#uninstall-assembly-from-cache)
- [Diagnosing network connectivity issues](#diagnosing-network-connectivity-issues)
- [.NET Core](#net-core)
- [.NET Framework](#net-framework)
- [ASP.NET Core](#aspnet-core)
- [Collecting ASP.NET Core logs](#collecting-aspnet-core-logs)
- [ILogger logs](#ilogger-logs)
- [DiagnosticSource logs](#diagnosticsource-logs)
- [Collecting ASP.NET Core performance counters](#collecting-aspnet-core-performance-counters)
- [ASP.NET \(.NET Framework\)](#aspnet-net-framework)
- [Examining ASP.NET process memory \(and dumps\)](#examining-aspnet-process-memory-and-dumps)
- [Profiling ASP.NET](#profiling-aspnet)
- [Application instrumentation](#application-instrumentation)
- [ASP.NET ETW providers](#aspnet-etw-providers)
- [Collect events using the Perfecto tool](#collect-events-using-the-perfecto-tool)
- [Collect events using FREB](#collect-events-using-freb)
<!-- /MarkdownTOC -->
## General .NET debugging tips
### Loading the SOS extension into WinDbg
When debugging a **.NET Framework application**, WinDbgX should automatically find a correct version of the SOS.dll. If it fails to do so and your .NET Framework version matches the one of the target app, use the following command:
```
.loadby sos mscorwks (.NET 2.0/3.5)
.loadby sos clr (.NET 4.0+)
```
For **.NET Core**, you need to download and install the **dotnet-sos** tool. The install command informs how to load SOS into WinDbg, for example:
```
> dotnet tool install -g dotnet-sos
...
> dotnet sos install
...
Execute '.load C:\Users\me\.dotnet\sos\sos.dll' to load SOS in your Windows debugger.
Cleaning up...
SOS install succeeded
```
SOS commands sometimes get overriden by other extensions help files. In such case, use **!sos.help \[cmd\]** command, for example, `!sos.help !savemodule`.
### Manually loading symbol files for .NET Core
I noticed that sometimes Microsoft public symbol servers do not have .NET Core dlls symbols. That does not allow WinDbg to decode native .NET stacks. Fortunately, we may solve this problem by precaching symbol files using the [dotnet-symbol](https://github.com/dotnet/symstore/tree/master/src/dotnet-symbol) tool. Assuming we set our `_NT_SYMBOL_PATH` to `SRV*C:\symbols\dbg*https://msdl.microsoft.com/download/symbols`, we need to run dotnet-symbol with the **--cache-directory** parameter pointing to our symbol cache folder (for example, `C:\symbols\dbg`):
```
dotnet-symbol --recurse-subdirectories --cache-directory c:\symbols\dbg -o C:\temp\toremove "C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.0.0\*"
```
We may later remove the `C:\temp\toremove` folder as all PDB files are indexed in the cache directory. The output folder contains both DLL and PDB files, takes lots of space, and is often not required.
### Disabling JIT optimization
For **.NET Core**, set the **COMPlus_JITMinOptsx** environment variable:
```
export COMPlus_JITMinOpts=1
```
For **.NET Framework**, you need to create an ini file. The ini file must have the same name as the executable with only extension changed to ini, eg. my.ini file will work with my.exe application.
```
[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0
```
### Decoding managed stacks in Sysinternals
As of version 16.22 version, **Process Explorer** understands managed stacks and should display them correctly when you double click on a thread in a process.
**Process Monitor**, unfortunately, lacks this feature. Pure managed modules will appear as `<unknown>` in the call stack view. However, we may fix the problem for the ngened assemblies. First, you need to generate a .pdb file for the ngened assembly, for example, `ngen createPDB c:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\e2c5db271896923f5450a77229fb2077\mscorlib.ni.dll c:\symbols\private`. Then make sure you have this path in your `_NT_SYMBOL_PATH` variable, for example, `C:\symbols\private;SRV*C:\symbols\dbg*http://msdl.microsoft.com/download/symbols`. If procmon still does not resolve the symbols, go to Options - Configure Symbols and reload the dbghelp.dll. I observe this issue in version 3.50.
### Check runtime version
For .NET Framework 2.0, you could check the version of mscorwks in the file properties or, if in debugger, using lmmv. For .NET Framework 4.x, you need to check clr.dll (or the Release value under the `HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full` key) and find it in the [Microsoft Docs](https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/versions-and-dependencies).
In .NET Core, we could run **dotnet --list-runtimes** command to list the available runtimes.
### Debugging/tracing a containerized .NET application (Docker)
With the introduction of EventPipes in .NET Core 2.1, the easiest approach is to create a shared `/tmp` volume and use a sidecar diagnostics container. A sample Dockerfile.netdiag may look as follows:
```
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS base
RUN apt-get update && apt-get install -y lldb; \
dotnet tool install -g dotnet-symbol; \
dotnet tool install -g dotnet-sos; \
/root/.dotnet/tools/dotnet-sos install
RUN dotnet tool install -g dotnet-counters; \
dotnet tool install -g dotnet-trace; \
dotnet tool install -g dotnet-dump; \
dotnet tool install -g dotnet-gcdump; \
echo 'export PATH="$PATH:/root/.dotnet/tools"' >> /root/.bashrc
ENTRYPOINT ["/bin/bash"]
```
You may use it to create a .NET diagnostics Docker image, for example:
```
$ docker build -t netdiag -f .\Dockerfile.netdiag .
```
Then, create a `/tmp` volume and mount it into your .NET application container, for example:
```
$ docker volume create dotnet-tmp
$ docker run --rm --name helloserver --mount "source=dotnet-tmp,target=/tmp" -p 13000:13000 helloserver 13000
```
And you are ready to run the diagnostics container and diagnose the remote application:
```
$ docker run --rm -it --mount "source=dotnet-tmp,target=/tmp" --pid=container:helloserver netdiag
root@d4bfaa3a9322:/# dotnet-trace ps
1 dotnet /usr/share/dotnet/dotnet
```
If you only want to trace the application with **dotnet-trace**, consider using a shorter Dockerfile.nettrace file:
```
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS base
RUN dotnet tool install -g dotnet-trace
ENTRYPOINT ["/root/.dotnet/tools/dotnet-trace", "collect", "-n", "dotnet", "-o", "/work/trace.nettrace", "@/work/input.rsp"]
```
where input.rsp:
```
--providers Microsoft-Windows-DotNETRuntime:0x14C14FCCBD:4,Microsoft-DotNETCore-SampleProfiler:0xF00000000000:4
```
The nettrace container will automatically start the tracing session enabling the providers from the input.rsp file. It also assumes the destination process name is dotnet:
```
$ docker build -t nettrace -f .\Dockerfile.nettrace .
$ docker run --rm --pid=container:helloserver --mount "source=dotnet-tmp,target=/tmp" -v "$pwd/:/work" -it nettrace
Provider Name Keywords Level Enabled By
Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --providers
Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --providers
Process : /usr/share/dotnet/dotnet
Output File : /work/trace.nettrace
[00:00:00:02] Recording trace 261.502 (KB)
Press <Enter> or <Ctrl+C> to exit...11 (KB)
Stopping the trace. This may take up to minutes depending on the application being traced.
```
## Diagnosing exceptions or erroneous behavior
### Using Time Travel Debugging (TTD)
Time Travel Debugging is an excellent way of troubleshooting errors and exceptions. We can step through the code causing the problems at our own pace. I describe TTD in [a WinDbg guide](/guides/windbg). It is my preferred way of debugging issues in applications and I highly recommend giving it a try.
### Collecting a memory dump
**[dotnet-dump](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-dump)** is one of the .NET diagnostics CLI tools. You may download it using curl or wget, for example: `curl -JLO https://aka.ms/dotnet-dump/win-x64`.
To create a full memory dump, run one of the commands:
```
dotnet-dump collect -p <process-id>
dotnet-dump collect -n <process-name>
```
You may create a heap-only memory dump by adding the **--type=Heap** option.
Createdump shares the location with the coreclr library, for example, for .NET 5: `/usr/share/dotnet/shared/Microsoft.NETCore.App/5.0.3/createdump` or `c:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.3\createdump.exe`.
To create a full memory dump, run **createdump --full {process-id}**. With no options provided, it creates a memory dump with heap memory, which equals to **createdump --withheap {pid}**.
The .NET application may run **createdump** automatically on crash. We configure this feature through [environment variables](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/collect-dumps-crash), for example:
```shell
# enable a memory dump creation on crash
set DOTNET_DbgEnableMiniDump=1
# when crashing, create a heap (2) memory dump, (4) for full memory dump
set DOTNET_DbgMiniDumpType=2
```
Apart from the .NET tools described above, you may create memory dumps with tools described in [the guide dedicated to diagnosing native Windows applications](diagnosing-native-windows-apps). As those tools usually do not understand .NET memory layout, I recommend creating full memory dumps to have all the necessary metadata for later analysis.
### Analysing exception information
First make sure with the **!Threads** command (SOS) that your current thread is the one with the exception context:
```
0:000> !Threads
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
0 1 1ec8 000000000055adf0 2a020 Preemptive 0000000002253560:0000000002253FD0 00000000004fb970 0 Ukn System.ArgumentException 0000000002253438
5 2 1c74 00000000005851a0 2b220 Preemptive 0000000000000000:0000000000000000 00000000004fb970 0 Ukn (Finalizer)
```
In the snippet above we can see that the exception was thrown on the thread no. 0 and this is our currently selected thread (in case it's not, we would use **\~0s** command) so we may use the **!PrintException** command from SOS (alias **!pe**), for example:
```
0:000> !pe
Exception object: 0000000002253438
Exception type: System.ArgumentException
Message: v should not be null
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80070057
```
To see the full managed call stack, use the **!CLRStack** command. By default, the debugger will stop on an unhandled exception. If you want to stop at the moment when an exception is thrown (first-chance exception), run the **sxe clr** command at the beginning of the debugging session.
## Diagnosing hangs
We usually start the analysis by looking at the threads running in a process. The call stacks help us identify blocked threads. We can use TTD, thread-time trace, or memory dumps to learn about what threads are doing. In the follow-up sections, I will describe how to find lock objects and relations between threads in memory dumps.
### Listing threads call stacks
To list native stacks for all the threads in **WinDbg**, run: **~\*k** or **~\*e!dumpstack**. If you are interested only in managed stacks, you may use the **~\*e!clrstack** SOS command. The **dotnet-dump**'s **analyze** command provides a super useful parallel stacks command:
```
> dotnet dump analyze test.dmp
> pstacks
________________________________________________
~~~~ 5cd8
1 System.Threading.Monitor.Enter(Object, Boolean ByRef)
1 deadlock.Program.Lock2()
~~~~ 3e58
1 System.Threading.Monitor.Enter(Object, Boolean ByRef)
1 deadlock.Program.Lock1()
2 System.Threading.Tasks.Task.InnerInvoke()
...
2 System.Threading.ThreadPoolWorkQueue.Dispatch()
2 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
```
In **LLDB**, we may show native call stacks for all the threads with the **bt all** command. Unfortunately, if we want to use !dumpstack or !clrstack commands, we need to manually switch between threads with the thread select command.
### Finding locks in managed code
You may examine thin locks using **!DumpHeap -thinlocks**. To find all sync blocks, use the **!SyncBlk -all** command.
On .NET Framework, you may also use the **!dlk** command from the SOSEX extension. It is pretty good in detecting deadlocks, for example:
```
0:007> .load sosex
0:007> !dlk
Examining SyncBlocks...
Scanning for ReaderWriterLock(Slim) instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Scanning for threads waiting on SyncBlocks...
Scanning for threads waiting on ReaderWriterLock locks...
Scanning for threads waiting on ReaderWriterLocksSlim locks...
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System\3a4f0a84904c4b568b6621b30306261c\System.ni.dll
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v4.0.30319_32\System.Transactions\ebef418f08844f99287024d1790a62a4\System.Transactions.ni.dll
Scanning for threads waiting on CriticalSections...
*DEADLOCK DETECTED*
CLR thread 0x1 holds the lock on SyncBlock 011e59b0 OBJ:02e93410[System.Object]
...and is waiting on CriticalSection 01216a58
CLR thread 0x3 holds CriticalSection 01216a58
...and is waiting for the lock on SyncBlock 011e59b0 OBJ:02e93410[System.Object]
CLR Thread 0x1 is waiting at clr!CrstBase::SpinEnter+0x92
CLR Thread 0x3 is waiting at System.Threading.Monitor.Enter(System.Object, Boolean ByRef)(+0x17 Native)
```
When debugging locks in code that is using tasks it is often necessary to examine execution contexts assigned to the running threads. I prepared a simple script which lists threads with their execution contexts. You only need (as in previous script) to find the MT of the Thread class in your appdomain, e.g.
```
0:036> !Name2EE mscorlib.dll System.Threading.Thread
Module: 72551000
Assembly: mscorlib.dll
Token: 020001d1
MethodTable: 72954960
EEClass: 725bc0c4
Name: System.Threading.Thread
```
And then paste it in the scripts below:
x86 version:
```
.foreach ($addr {!DumpHeap -short -mt <METHODTABLE> }) { .printf /D "Thread: %i; Execution context: <link cmd=\"!do %p\">%p</link>\n", poi(${$addr}+28), poi(${$addr}+8), poi(${$addr}+8) }
```
x64 version:
```
.foreach ($addr {!DumpHeap -short -mt <METHODTABLE> }) { .printf /D "Thread: %i; Execution context: <link cmd=\"!do %p\">%p</link>\n", poi(${$addr}+4c), poi(${$addr}+10), poi(${$addr}+10) }
```
Notice that the thread number from the output is a managed thread id and to map it to the windbg thread number you need to use the !Threads command.
## Diagnosing waits or high CPU usage
Dotnet-trace allows us to enable the runtime CPU sampling provider (**Microsoft-DotNETCore-SampleProfiler**). However, using it might impact application performance as it internally calls **ThreadSuspend::SuspendEE** to suspend managed code execution while collecting the samples. Although it is a sampling profiler, it is a bit special. It runs on a separate thread and collects stacks of all the managed threads, even the waiting ones. This behavior resembles the thread time profiler. Probably that's the reason why PerfView shows us the **Thread Time** view when opening the .nettrace file.
Sample collect examples:
```bash
dotnet-trace collect --profile cpu-sampling -p 12345
dotnet-trace collect --profile cpu-sampling -- myapp.exe
```
Dotnet-trace does not automatically enable DiagnosticSource or TPL providers. Therefore, if we want to see activities in PerfView, we need to turn them on manually, for example:
```bash
dotnet-trace collect --profile cpu-sampling --providers "Microsoft-Diagnostics-DiagnosticSource:0xFFFFFFFFFFFFF7FF:4:FilterAndPayloadSpecs=HttpHandlerDiagnosticListener/System.Net.Http.Request@Activity2Start:Request.RequestUri\nHttpHandlerDiagnosticListener/System.Net.Http.Response@Activity2Stop:Response.StatusCode,System.Threading.Tasks.TplEventSource:1FF:5" -n testapp
```
For diagnosing CPU problems in .NET applications running on Windows, we may also rely on ETW (Event Tracing for Windows). In [a guide dedicated to diagnosing native applications](diagnosing-native-windows-apps), I describe how to collect and analyze ETW traces.
On Linux, we additionally have the [perfcollect](https://docs.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng) script. It is the easiest way to use Linux Kernel perf_events for diagnosing .NET apps. In my tests, however, I found that quite often, it did not correctly resolve .NET stacks.
To collect CPU samples with perfcollect, use the **perfcollect collect** command. To also enable the Thread Time events, add the **-threadtime** option. If only possible, I would recommend opening the traces (even the ones from Linux) in PerfView. But if it's impossible, try the **view** command of the perfcollect script, for example:
```bash
perfcollect view sqrt.trace.zip -graphtype caller
```
Using the **-graphtype** option, we may switch from the top-down view (`caller`) to the bottom-up view (`callee`).
## Diagnosing managed memory leaks
### Collecting memory snapshots
If we are interested only in GC Heaps, we may create the GC Heap snapshot using **PerfView**:
perfview heapsnapshot <pid|name>
In GUI, we may use the menu option: **Memory -> Take Heap Snapshot**.
For .NET Core applications, we have a CLI tool: **dotnet-gcdump**, which you may get from the https://aka.ms/dotnet-gcdump/runtime-id URL, for example, https://aka.ms/dotnet-gcdump/linux-x64. And to collect the GC dump we need to run one of the commands:
```
dotnet-gcdump -p <process-id>
dotnet-gcdump -n <process-name>
```
Sometimes managed heap is not enough to diagnose the memory leak. In such situations, we need to create a memory dump, as described in [a guide dedicated to diagnosing native applications](diagnosing-native-windows-apps).
### Analyzing collected snapshots
**PerfView** can open GC Heap snapshots and dumps. If you only have a memory dump, you may convert a memory dump file to a PerfView snapshot using **PerfView HeapSnapshotFromProcessDump ProcessDumpFile {DataFile}** or using the GUI options **Memory -> Take Heap Snapshot from Dump**.
I would like to bring your attention to an excellent diffing option available for heap snapshots. Imagine you made two heap snapshots of the leaking process:
- first named LeakingProcess.gcdump
- second (taken a minute later) named LeakingProcess.1.gcdump
You may now run PerfView, open two collected snapshots, switch to the LeakingProcess.1.gcdump and under the Diff menu you should see an option to diff this snapshot with the baseline:
After you choose it, a new window will pop up with a tree of objects which have changed between the snapshots. Of course, if you have more snapshots you can generate diffs between them all. A really powerful feature!
**WinDbg** allows you to analyze the full memory dumps. **Make sure that bitness of the dump matches bitness of the debugger.** Then load the SOS extension and identify objects which use most of the memory using **!DumpHeap -stat**. Later, analyze the references using the **!GCRoot** command.
Other SOS commands for analyzing the managed heap include:
```
!EEHeap [-gc] [-loader]
!HeapStat [-inclUnrooted | -iu]
!DumpHeap [-stat]
[-strings]
[-short]
[-min <size>]
[-max <size>]
[-live]
[-dead]
[-thinlock]
[-startAtLowerBound]
[-mt <MethodTable address>]
[-type <partial type name>]
[start [end]]
!ObjSize [<Object address>]
!GCRoot [-nostacks] <Object address>
!DumpObject <address> | !DumpArray <address> | !DumpVC <mt> <address>
```
**dotnet-gcdump** has a **report** command that lists the objects recorded in the GC heaps. The output resembles output from the SOS `!dumpheap` command.
## Diagnosing issues with assembly loading
### Troubleshooting loading with EventPipes/ETW (.NET)
The **Loader** keyword (`0x8`) in the **Microsoft-Windows-DotNETRuntime** provider enables events relating to **loading and unloading** of **appdomains**, **assemblies** and **modules**.
Starting with **.NET 5**, the new **AssemblyLoader** keyword (`0x4`) gives us a detailed view of the **assembly resolution process**. Additionally, we can group the activity events per assembly using the `ActivityID`.
dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:C -- testapp.exe
### Troubleshooting loading using ETW (.NET Framework)
There is a number of ETW events defined under the **Microsoft-Windows-DotNETRuntimePrivate/Binding/** category. We may use, for example, **PerfView** to collect them. Just make sure that you have the .NET check box selected in the collection dialog. Start collection and stop it after the loading exception occurs. Then open the .etl file, go to the **Events** screen and filter them by *binding*. Select all of the events and press ENTER. PerfView will immediately print the instances of the selected events in the grid on the right. You may later search or filter the grid with the help of the search boxes above it.
### Troubleshooting loading using Fusion log (.NET Framework)
Fusion log is available in all versions of the .NET Framework. There is a tool named **fuslogvw** in .NET SDK, which you may use to set the Fusion log configuration. Andreas Wäscher implemented an easier-to-use version of this tool, with a modern UI, named [Fusion++](https://github.com/awaescher/Fusion). You may download the precompiled version from the [release page](https://github.com/awaescher/Fusion/releases/).
If using neither of the above tools is possible (for example, you are in a restricted environment), you may configure the Fusion log through **registry settings**. The root of all the Fusion log settings is **HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion**.
When writing to a folder on a hard drive fusion logs are split among categories and processes, e.g.:
```
C:\TEMP\FUSLOGVW
├───Default
│ └───powershell.exe
└───NativeImage
└───powershell.exe
```
Log to exception text:
HKEY_LOCAL_MACHINE\software\microsoft\fusion
EnableLog REG_DWORD 0x1
or
reg delete HKLM\Software\Microsoft\Fusion /va
reg add HKLM\Software\Microsoft\Fusion /v EnableLog /t REG_DWORD /d 0x1
Log failures to disk:
HKEY_LOCAL_MACHINE\software\microsoft\fusion
LogFailures REG_DWORD 0x1
LogPath REG_SZ c:\logs\fuslogvw
or
reg delete HKLM\Software\Microsoft\Fusion /va
reg add HKLM\Software\Microsoft\Fusion /v LogFailures /t REG_DWORD /d 0x1
reg add HKLM\Software\Microsoft\Fusion /v LogPath /t REG_SZ /d "C:\logs\fuslogvw"
Log all binds to disk
HKEY_LOCAL_MACHINE\software\microsoft\fusion
LogPath REG_SZ c:\logs\fuslogvw
ForceLog REG_DWORD 0x1
or
reg delete HKLM\Software\Microsoft\Fusion /va
reg add HKLM\Software\Microsoft\Fusion /v ForceLog /t REG_DWORD /d 0x1
reg add HKLM\Software\Microsoft\Fusion /v LogPath /t REG_SZ /d "C:\logs\fuslogvw"
Log disabled
HKEY_LOCAL_MACHINE\software\microsoft\fusion
LogPath REG_SZ c:\logs\fuslogvw
or
reg delete HKLM\Software\Microsoft\Fusion /va
### GAC (.NET Framework)
For .NET2.0/3.5 Global Assembly Cache was located in **c:\Windows\assembly** folder with a drag/drop option for installing/uninstalling assemblies. Citing [a stackoverflow answer](http://stackoverflow.com/questions/10013047/gacutil-vs-manually-editing-c-windows-assembly):
> This functionality is provided by a custom shell extension, shfusion.dll. It flattens the GAC and makes it look like a single folder. And takes care of automatically un/registering the assemblies for you when you manipulate the explorer window. So you’re fine doing this.
To **disable GAC viewer in Windows Explorer**, add a DWORD value **DisableCacheViewer** set to 1 under the **HKLM\Software\Microsoft\Fusion** key.
Note that this will no longer work for .NET 4, it uses in a different folder to store GAC files (**c:\windows\microsoft.net\assembly**) and that folder does not have the same kind of shell extension. Thus, you can see the raw content of it. However, you should not directly use it.
It is best to use **gacutil** to manipulate GAC content. Though it’s possible to install assembly in both GAC folders as stated [here](http://stackoverflow.com/questions/7095887/registering-the-same-version-of-an-assembly-but-with-different-target-frameworks), but I would not consider it a good practice as framework tools can’t deal with it. .NET GAC settings are stored under the registry key: HKLM\Software\Microsoft\Fusion.
#### Find assembly in cache
We can use the **gacutil /l** to find an assembly in GAC. If no name is provided, the command lists all the assemblies in cache.
gacutil /l System.Core
The Global Assembly Cache contains the following assemblies:
System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL
System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL
Number of items = 2
#### Uninstall assembly from cache
gacutil /u MyTest.exe
## Diagnosing network connectivity issues
### .NET Core
.NET Core provides a number of ETW and EventPipes providers to collect the network tracing events. Enabling the providers could be done in **dotnet-trace**, **PerfView**, or **dotnet-wtrace**. Network ETW providers use only two keywords (`Default = 0x1` and `Debug = 0x2`) and, as usual, we may filter the events by the log level (from 1 (critical) to 5 (verbose)).
In **.NET 5**, the providers were renamed and currently we can use the following names:
- `Private.InternalDiagnostics.System.Net.Primitives` - cookie container, cache credentials logs
- `Private.InternalDiagnostics.System.Net.Sockets` - logs describing operations on sockets, connection status events,
- `Private.InternalDiagnostics.System.Net.NameResolution`
- `Private.InternalDiagnostics.System.Net.Mail`
- `Private.InternalDiagnostics.System.Net.Requests` - logs from System.Net.Requests classes
- `Private.InternalDiagnostics.System.Net.HttpListener`
- `Private.InternalDiagnostics.System.Net.WinHttpHandler`
- `Private.InternalDiagnostics.System.Net.Http` - HttpClient and HTTP handler logs, authentication events
- `Private.InternalDiagnostics.System.Net.Security` - SecureChannel (TLS) events, Windows SSPI logs
For previous .NET Core versions, the names were as follows:
- `Microsoft-System-Net-Primitives`
- `Microsoft-System-Net-Sockets`
- `Microsoft-System-Net-NameResolution`
- `Microsoft-System-Net-Mail`
- `Microsoft-System-Net-Requests`
- `Microsoft-System-Net-HttpListener`
- `Microsoft-System-Net-WinHttpHandler`
- `Microsoft-System-Net-Http`
- `Microsoft-System-Net-Security`
We may create a network.rsp file that enables all these event sources and the Kestrel one. You may use it with **dotnet-trace**, for example:
```
$ dotnet-trace collect -n dotnet @network.rsp
```
The network.rsp file for older .NET Core (before .NET 5) might look as follows:
```
--providers Microsoft-System-Net-Primitives,Microsoft-System-Net-Sockets,Microsoft-System-Net-NameResolution,Microsoft-System-Net-Mail,Microsoft-System-Net-Requests,Microsoft-System-Net-HttpListener,Microsoft-System-Net-WinHttpHandler,Microsoft-System-Net-Http,Microsoft-System-Net-Security,Microsoft-AspNetCore-Server-Kestrel
```
For .NET 5 and newer:
```
--providers
Private.InternalDiagnostics.System.Net.Primitives,Private.InternalDiagnostics.System.Net.Sockets,Private.InternalDiagnostics.System.Net.NameResolution,Private.InternalDiagnostics.System.Net.Mail,Private.InternalDiagnostics.System.Net.Requests,Private.InternalDiagnostics.System.Net.HttpListener,Private.InternalDiagnostics.System.Net.WinHttpHandler,Private.InternalDiagnostics.System.Net.Http,Private.InternalDiagnostics.System.Net.Security,Microsoft-AspNetCore-Server-Kestrel
```
I also developed [**dotnet-wtrace**](https://github.com/lowleveldesign/dotnet-wtrace), a lightweight traces that makes it straightfoward to live collect .NET events, including network traces.
### .NET Framework
All classes from `System.Net`, if configured properly, may provide a lot of interesting logs through the default System.Diagnostics mechanisms. The list of the available trace sources is available in [Microsoft docs](https://docs.microsoft.com/en-us/dotnet/framework/network-programming/how-to-configure-network-tracing).
This is a configuration sample which writes network traces to a file:
```xml
<system.diagnostics>
<trace autoflush="true" />
<sharedListeners>
<add name="file" initializeData="C:\logs\network.log" type="System.Diagnostics.TextWriterTraceListener" />
</sharedListeners>
<sources>
<source name="System.Net" switchValue="Verbose" tracemode="includehex" maxdatasize="512">
<listeners>
<add name="file" />
</listeners>
</source>
<source name="System.Net.Http" switchValue="Verbose">
<listeners>
<add name="file" />
</listeners>
</source>
<source name="System.Net.HttpListener" switchValue="Verbose">
<listeners>
<add name="file" />
</listeners>
</source>
<source name="System.Net.Sockets" switchValue="Verbose">
<listeners>
<add name="file" />
</listeners>
</source>
</sources>
</system.diagnostics>
```
These logs may be verbose and numerous, therefore, I suggest starting with Information level and smaller number of sources. You may also consider using **EventProviderTraceListener** to make the trace writes faster and less impactful. An example configuration file with those changes:
```xml
<system.diagnostics>
<trace autoflush="true" />
<sharedListeners>
<add name="etw" initializeData="{0f09a664-1713-4665-91e8-8d6b8baee030}" type="System.Diagnostics.Eventing.EventProviderTraceListener, System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
</sharedListeners>
<sources>
<source name="System.Net" switchValue="Information">
<listeners>
<add name="etw" />
</listeners>
</source>
<source name="System.Net.Http" switchValue="Information">
<listeners>
<add name="etw"/>
</listeners>
</source>
<source name="System.Net.HttpListener" switchValue="Information">
<listeners>
<add name="etw"/>
</listeners>
</source>
</sources>
</system.diagnostics>
```
And to collect such a trace:
```shell
logman start "net-trace-session" -p "{0f09a664-1713-4665-91e8-8d6b8baee030}" -bs 512 -nb 8 64 -o "c:\temp\net-trace.etl" -ets & pause & logman stop net-trace-session -ets
```
## ASP.NET Core
### Collecting ASP.NET Core logs
For low-level network traces, you may enable .NET network providers, as described in the previous section. ASP.NET Core framework logs events either through **DiagnosticSource** using **Microsoft.AspNetCore** as the source name or through the **ILogger** interface.
#### ILogger logs
The CreateDefaultBuilder method adds LoggingEventSource (named **Microsoft-Extensions-Logging**) as one of the log outputs. The **FilterSpecs** argument makes it possible to filter the events by logger name and level, for example:
```
Microsoft-Extensions-Logging:5:5:FilterSpecs=webapp.Pages.IndexModel:0
```
We may define the log message format with keywords (pick one):
- 0x1 - enable meta events
- 0x2 - enable events with raw arguments
- 0x4 - enable events with formatted message (the most readable)
- 0x8 - enable events with data seriazlied to JSON
For example, to collect ILogger info messages: `dotnet-trace collect -p PID --providers "Microsoft-Extensions-Logging:0x4:0x4"`
#### DiagnosticSource logs
To listen to **DiagnosticSource events**, we should enable the **Microsoft-Diagnostics-DiagnosticSource** event source. DiagnosticSource events often contain complex types and we need to use [parser specifications](https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/DiagnosticSourceEventSource.cs) to extract the interesting properties.
The **Microsoft-Diagnostics-DiagnosticSourcex** event source some special keywords:
- 0x1 - enable diagnostic messages
- 0x2 - enable regular events
- 0x0800 - disable the shortcuts keywords, listed below
- 0x1000 - enable activity tracking and basic hosting events (ASP.NET Core)
- 0x2000 - enable activity tracking and basic command events (EF Core)
Also, we should enable the minimal logging from the **System.Threading.Tasks.TplEventSource** provider to profit from the [activity tracking](https://docs.microsoft.com/en-us/archive/blogs/vancem/exploring-eventsource-activity-correlation-and-causation-features).
When our application is hosted on the Kestrel server, we may enable the **Microsoft-AspNetCore-Server-Kestrel** provider to get Kestrel events.
An example command that enables all ASP.NET Core event traces and some other useful network event providers. It also adds activity tracking for **HttpClient** requests:
```
> dotnet-trace collect --providers "Private.InternalDiagnostics.System.Net.Security,Private.InternalDiagnostics.System.Net.Sockets,Microsoft-AspNetCore-Server-Kestrel,Microsoft-Diagnostics-DiagnosticSource:0x1003:5:FilterAndPayloadSpecs=\"Microsoft.AspNetCore\nHttpHandlerDiagnosticListener\nHttpHandlerDiagnosticListener/System.Net.Http.Request@Activity2Start:Request.RequestUri\nHttpHandlerDiagnosticListener/System.Net.Http.Response@Activity2Stop:Response.StatusCode\",System.Threading.Tasks.TplEventSource:0x80:4,Microsoft-Extensions-Logging:4:5" -n webapp
```
### Collecting ASP.NET Core performance counters
ASP.NET Core provides some basic performance counters through the **Microsoft.AspNetCore.Hosting** event source. If we are also using Kestrel, we may add some interesting counters by enabling **Microsoft-AspNetCore-Server-Kestrel**:
```
> dotnet-counters monitor "Microsoft.AspNetCore.Hosting" "Microsoft-AspNetCore-Server-Kestrel" -n testapp
Press p to pause, r to resume, q to quit.
Status: Running
[Microsoft.AspNetCore.Hosting]
Current Requests 0
Failed Requests 0
Request Rate (Count / 1 sec) 0
Total Requests 0
[Microsoft-AspNetCore-Server-Kestrel]
Connection Queue Length 0
Connection Rate (Count / 1 sec) 0
Current Connections 1
Current TLS Handshakes 0
Current Upgraded Requests (WebSockets) 0
Failed TLS Handshakes 2
Request Queue Length 0
TLS Handshake Rate (Count / 1 sec) 0
Total Connections 7
Total TLS Handshakes 7
```
## ASP.NET (.NET Framework)
### Examining ASP.NET process memory (and dumps)
Some useful [PSSCOR4](http://www.microsoft.com/en-us/download/details.aspx?id=21255) commands for ASP.NET:
```
!ProcInfo [-env] [-time] [-mem]
FindDebugTrue
!FindDebugModules [-full]
!DumpHttpContext dumps the HttpContexts in the heap. It shows the status of the request and the return code, etc. It also prints out the start time
!ASPXPages just calls !DumpHttpContext to print out information on the ASPX pages running on threads.
!DumpASPNETCache [-short] [-stat] [-s]
!DumpRequestTable [-a] [-p] [-i] [-c] [-m] [-q] [-n] [-e] [-w] [-h] [-r] [-t] [-x] [-dw] [-dh] [-de] [-dx]
!DumpHistoryTable [-a]
!DumpHistoryTable dumps the aspnet_wp history table.
!DumpBuckets dumps entire request table buckets.
!GetWorkItems given a CLinkListNode, print out request & work items.
```
[Netext](http://netext.codeplex.com/) commands for ASP.NET:
```
!whttp [/order] [/running] [/withthread] [/status <decimal>] [/notstatus <decimal>] [/verb <string>] [<expr>] - dump HttpContext objects
!wconfig - dump configuration sections loaded into memory
!wruntime - dump all active Http Runtime information
```
### Profiling ASP.NET
### Application instrumentation
Interesting tools and libraries:
- [ASP.NET 4.5 page instrumentation mechanism - PageExecutionListener](http://weblogs.asp.net/imranbaloch/archive/2013/11/23/page-instrumentation-in-asp-net-4-5.aspx)
- [Glimpse](https://github.com/glimpse/glimpse)
- [MiniProfiler](https://miniprofiler.com/)
- [Elmah](https://elmah.github.io/)
We may also use the ASP.NET trace listener to print diagnostic message to the page trace. In the configuration file below, we configure the Performance TraceSource to pass events to the ASP.NET trace listener.
```xml
<?xml version="1.0"?>
<configuration>
<system.diagnostics>
<trace autoflush="true" />
<sharedListeners>
<add name="WebPageTraceListener" type="System.Web.WebPageTraceListener, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
</sharedListeners>
<sources>
<source name="Performance" switchValue="Verbose">
<listeners>
<add name="WebPageTraceListener" />
</listeners>
</source>
</sources>
</system.diagnostics>
<system.web>
<trace enabled="true" localOnly="true" pageOutput="false" />
<customErrors mode="Off">
</customErrors>
<compilation debug="true" targetFramework="4.0">
</compilation>
</system.web>
</configuration>
```
### ASP.NET ETW providers
ASP.NET ETW providers are defined in the aspnet.mof file in the main .NET Framework folder. They should be installed with the framework:
```
> logman query /providers "ASP.NET Events"
Provider GUID
-------------------------------------------------------------------------------
ASP.NET Events {AFF081FE-0247-4275-9C4E-021F3DC1DA35}
Value Keyword Description
-------------------------------------------------------------------------------
0x0000000000000001 Infrastructure Infrastructure Events
0x0000000000000002 Module Pipeline Module Events
0x0000000000000004 Page Page Events
0x0000000000000008 AppServices Application Services Events
Value Level Description
-------------------------------------------------------------------------------
0x01 Fatal Abnormal exit or termination
0x02 Error Severe errors
0x03 Warning Warnings
0x04 Information Information
0x05 Verbose Detailed information
```
If they are not, use mofcomp.exe to install them.
To start collecting trace events from the ASP.NET and IIS providers run the following command:
```
logman start aspnettrace -pf ctrl-iis-aspnet.guids -ct perf -o aspnet.etl -ets
```
where the ctrl-iis-aspnet.guids looks as follows:
```
{AFF081FE-0247-4275-9C4E-021F3DC1DA35} 0xf 5 ASP.NET Events
{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83} 0x1ffe 5 IIS: WWW Server
```
And stop it with the command:
```
logman stop aspnettrace -ets
```
### Collect events using the Perfecto tool
Perfecto is a tool that creates an ASP.NET data collector in the system and allows you to generate nice reports of requests made to your ASP.NET application. After installing you can either use the **perfmon** to start the report generation:
1. On perfmon, navigate to the "Performance\Data Collector Sets\User Defined\ASPNET Perfecto" node.
2. Click the "Start the Data Collector Set" button on the tool bar.
3. Wait for/or make requests to the server (more than 10 seconds).
4. Click the "Stop the Data Collector Set" button on the tool bar.
5. Click the "View latest report" button on the tool bar or navigate to the last report at "Performance\Reports\User Defined\ASPNET Perfecto"
or **logman**:
```
logman.exe start -n "Service\ASPNET Perfecto"
logman.exe stop -n "Service\ASPNET Perfecto"
```
Note: The View commands are also available as toolbar buttons.
Sometimes you can see an error like below:
```
Error Code: 0xc0000bf8
Error Message: At least one of the input binary log files contain fewer than two data samples.
```
This usually happens when you collected data too fast. The performance counters are set by default to collect every 10 seconds. So a fast start/stop sequence may end without enough counter data being collected. Always allow more than 10 seconds between a start and stop commands. Or otherwise delete the performance counters collector or change the sample interval.
Requirements:
1. Windows >= Vista
2. Installed IIS tracing (`dism /online /enable-feature /featurename:IIS-HttpTracing`)
### Collect events using FREB
New IIS servers (7.0 up) contain a nice diagnostics functionality called Failed Request Tracing (or **FREB**). You may find a lot of information how to enable it on the [IIS official site](https://www.iis.net/learn/troubleshoot/using-failed-request-tracing/troubleshooting-failed-requests-using-tracing-in-iis) and in my [iis debugging recipe](asp.net/troubleshooting-iis.md).
{% endraw %}
================================================
FILE: guides/diagnosing-native-windows-apps.md
================================================
---
layout: page
title: Diagnosing native Windows applications
date: 2025-05-25 08:00:00 +0200
---
{% raw %}
**Table of contents:**
<!-- MarkdownTOC -->
- [Debugging process execution](#debugging-process-execution)
- [Collecting memory dumps on errors](#collecting-memory-dumps-on-errors)
- [Using procdump](#using-procdump)
- [Using Windows Error Reporting \(WER\)](#using-windows-error-reporting-wer)
- [Automatic dump collection using AeDebug registry key](#automatic-dump-collection-using-aedebug-registry-key)
- [Diagnosing waits or high CPU usage](#diagnosing-waits-or-high-cpu-usage)
- [Collecting ETW trace](#collecting-etw-trace)
- [Anaysing the collected traces](#anaysing-the-collected-traces)
- [Diagnosing issues with DLL loading](#diagnosing-issues-with-dll-loading)
- [Diagnosing window functions \(user32\)](#diagnosing-window-functions-user32)
<!-- /MarkdownTOC -->
Debugging process execution
---------------------------
Please check [the WinDbg guide](/guides/windbg) where I describe various troubleshooting commands in WinDbg, along with Time Travel Debugging.
Collecting memory dumps on errors
---------------------------------
### Using procdump
My preferred tool to collect memory dumps is **[procdump](https://learn.microsoft.com/en-us/sysinternals/downloads/procdump)**.
It is often a good way to start diagnosing errors by observing 1st chance exceptions occurring in a process. At this point we don't want to collect any dumps, only logs. We may achieve this by specyfing a non-existing exception name in the filter command, for example:
```
C:\Utils> procdump -e 1 -f "DoesNotExist" 8012
...
CLR Version: v4.0.30319
[09:03:27] Exception: E0434F4D.System.NullReferenceException ("Object reference not set to an instance of an object.")
[09:03:28] Exception: E0434F4D.System.NullReferenceException ("Object reference not set to an instance of an object.")
```
We may also observe the logs in procmon. In order to see the procdump log events in **procmon** remember to add procdump.exe and procdump64.exe to the accepted process names in procmon filters.
To create a full memory dump when `NullReferenceException` occurs use the following command:
```
procdump -ma -e 1 -f "E0434F4D.System.NullReferenceException" 8012
```
From some time procdump uses a managed debugger engine when attaching to .NET Framework processes. This is great because we can filter exceptions based on their managed names. Unfortunately, that works only for 1st chance exceptions (at least for .NET 4.0). 2nd chance exceptions are raised out of the .NET Framework and must be handled by a native debugger. Starting from .NET 4.0 it is no longer possible to attach both managed and native engine to the same process. Thus, if we want to make a dump on the 2nd chance exception for a .NET application, we need to use the **-g** option in order to force procdump to use the native engine.
### Using Windows Error Reporting (WER)
By default WER takes dump only when necessary, but this behavior can be configured and we can force WER to always create a dump by modifying `HKLM\Software\Microsoft\Windows\Windows Error Reporting\ForceQueue=1` or (`HKEY_CURRENT_USER\Software\Microsoft\Windows\Windows Error Reporting\ForceQueue=1`). The reports are usually saved at `%LocalAppData%\Microsoft\Windows\WER`, in two directories: `ReportArchive`, when a server is available or `ReportQueue`, when the server is unavailable. If you want to keep the data locally, just set the server to a non-existing machine (for example, `HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\CorporateWERServer=NonExistingServer`). For **system processes** you need to look at `C:\ProgramData\Microsoft\Windows\WER`. In Windows 2003 Server R2 Error Reporting stores errors in the signed-in user's directory (for example, `C:\Documents and Settings\me\Local Settings\Application Data\PCHealth\ErrorRep`).
Starting with Windows Server 2008 and Windows Vista with Service Pack 1 (SP1), Windows Error Reporting can be configured to [collect full memory dumps on application crash](https://learn.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps). The registry key enabling this behavior is `HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps`. An example configuration for saving full-memory dumps to the %SYSTEMDRIVE%\dumps folder when the test.exe application fails looks as follows:
```
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\test.exe]
"DumpFolder"=hex(2):25,00,53,00,59,00,53,00,54,00,45,00,4d,00,44,00,52,00,49,\
00,56,00,45,00,25,00,5c,00,64,00,75,00,6d,00,70,00,73,00,00,00
"DumpType"=dword:00000002
```
With the help of [the WER API](https://learn.microsoft.com/en-us/windows/win32/wer/wer-reference), you may also force WER reports in your custom application or even
[register a custom crash handler](https://minidump.net/windows-error-reporting/).
To **completely disable WER**, create a DWORD Value under the `HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting` key, named `Disabled` and set its value to `1`. For 32-bit apps use the `HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\Windows Error Reporting` key.
### Automatic dump collection using AeDebug registry key
There is a special [AeDebug](https://learn.microsoft.com/en-us/windows/win32/debug/configuring-automatic-debugging) key in the registry defining what should happen when an unhandled exception occurs in an application. You may find it under the `HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion` key (or `HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion` for 32-bit apps). Its important value keys include:
- `Debugger` : REG_SZ - application which will be called to handle the problematic process (example value: `procdump.exe -accepteula -j "c:\dumps" %ld %ld %p`), the first %ld parameter is replaced with the process ID and the second with the event handle
- `Auto` : REG_SZ - defines if the debugger runs automatically, without prompting the user (example value: 1)
- `UserDebuggerHotKey` : REG_DWORD - not sure, but it looks it enables the Debug button on the exception handling message box (example value: 1)
To set **WinDbg** as your default AeDebug debugger, run `windbg -I`. After running this command, WinDbg will launch on application crashes. You may also automate WinDbg to create a memory dump and then allow process to terminate, for example: `windbg -c ".dump /ma /u c:\dumps\crash.dmp; qd" -p %ld -e %ld -g`.
My favourite tool to use as the automatic debugger is **procdump**. The command line to install it is `procdump -mp -i c:\dumps`, where c:\dumps is the folder where I would like to store the dumps of crashing apps.
Diagnosing waits or high CPU usage
----------------------------------
There are two ways of tracing CPU time. We could either use CPU sampling or Thread Time profiling. CPU sampling is about collecting samples in intervals: each CPU sample contains an instruction pointer to the currently executing code. Thus, this technique is excellent when diagnosing high CPU usage of an application. It won't work for analyzing waits in the applications. For such scenarios, we should rely on Thread Time profiling. It uses the system scheduler/dispatcher events to get detailed information about application CPU time. When combined with CPU sampling, it is the best non-invasive profiling solution.
### Collecting ETW trace
We may use **PerfView** or **wpr.exe** to collect CPU samples and Thread Time events.
When collecting CPU samples, PerfView relies on Profile events coming from the Kernel ETW provider which has very low impact on the system overall performance. An example command to start the CPU sampling:
```shell
perfview collect -NoGui -KernelEvents:Profile,ImageLoad,Process,Thread -ClrEvents:JITSymbols cpu-collect.etl
```
Alternatively, you may use the Collect dialog. Make sure the Cpu Samples checkbox is selected.
To collect Thread Time events, you may use the following command:
```shell
perfview collect -NoGui -ThreadTime thread-time-collect.etl
```
The Collect dialog has also the Thread Time checkbox.
### Anaysing the collected traces
For analyzing **CPU Samples**, use the **CPU Stacks** view. Always check the number of samples if it corresponds to the tracing time (CPU sampling works when we have enough events). If necessary, zoom into the interesting period using a histogram (select the time and press Alt + R). Checking the **By Name** tab could be enough to find the method responsible for the high CPU Usage (look at the inclusive time and make sure you use correct grouping patterns).
When analyzing waits in an application, we should use the **Thread Time Stacks** views. The default one, **with StartStop activities**, tries to group the tasks under activities and helps diagnose application activities, such as HTTP requests or database queries. Remember that the exclusive time in the activities view is a sum of all the child tasks. The thread under the activity is the thread on which the task started, not necessarily the one on which it continued. The **with ReadyThread** view can help when we are looking for thread interactions. For example, we want to find the thread that released a lock on which a given thread was waiting. The **Thread Time Stacks** view (with no grouping) is the best one to visualize the application's sequence of actions. Expanding thread nodes in the CallTree could take lots of time, so make sure you use other events (for example, from the Events view) to set the time ranges. As usual, check the grouping patterns.
Diagnosing issues with DLL loading
----------------------------------
An invaluable source of information when dealing with DLL loading issues are Windows Loader snaps. Those are detailed logs of the steps that Windows Loader takes to resolve the application library dependencies. They are one of the available Global Flags that we can set for an executable, so we may use the **gflags.exe** tool to enable them.
Alternatively, you may modify the process IFEO registry key, for example:
```
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\winver.exe]
"GlobalFlag"=dword:000000002
```
Once enabled, you need to start the failing application under a debugger and the Loader logs should appear in the debug output.
Alternatively, you may collect a procmon or ETW trace and search for any failure in the file events.
Diagnosing window functions (user32)
------------------------------------
The code snippet below contains example commands creating breakpoints to trace window functions:
```cpp
# 32-bit
bp user32!NtUserSetWindowPos ".printf \"SetWindowPos( hWnd: %p, hWndInsertAfter: %p, X: %d, Y: %d, cx: %d, cy: %d, uFlags: %x )\\n\", poi(@esp+4), poi(@esp+8), poi(@esp+0xC), poi(@esp+0x10), poi(@esp+0x14), poi(@esp+0x18), poi(@esp+0x1C); g"
bp user32!NtUserShowWindow ".printf \"ShowWindow( hWnd: %p, nCmdShow: %d )\\n\", poi(@esp+4), poi(@esp+8); g"
bp user32!SetWindowLongW ".printf \"SetWindowLongW( hWnd: %p, nIndex: %d, dwNewLong: %p )\\n\", poi(@esp+4), poi(@esp+8), poi(@esp+0xC); g"
bp user32!SetForegroundWindow ".printf \"SetForegroundWindow( hWnd: %p )\\n\", poi(@esp+4); g"
bp user32!NtUserSetParent ".printf \"SetParent( hWndChild: %p, hWndNewParent: %p )\\n\", poi(@esp+4), poi(@esp+8); g"
# 32-bit, but using dx
bp user32!NtUserSetWindowPos "dx new { function = \"SetWindowPos\", hWnd = *(void **)(@esp+4), hWndInsertAfter = *(void **)(@esp+8), X = *(int *)(@esp+0xC), Y = *(int *)(@esp+0x10), cx = *(int *)(@esp+0x14), cy = *(int *)(@esp+0x18), uFlags = *(unsigned int *)(@esp+0x1C) }; g"
bp user32!NtUserSetForegroundWindow "dx new { function = \"SetForegroundWindow\", hWnd = *(void **)(@esp+4) }; g"
bp user32!NtUserShowWindow "dx new { function = \"ShowWindow\", hWnd = *(void **)(@esp+4), nCmdShow = *(int *)(@esp+8) }; g"
bp user32!NtUserSetParent "dx new { function = \"SetParent\", hWndChild = *(void **)(@esp+4), hWndNewParent = *(void **)(@esp+8) }; g"
bp user32!NtUserSetWindowLong "dx new { function = \"SetWindowLongW\", hWnd = *(void **)(@esp+4), nIndex = *(int *)(@esp+8), dwNewLong = *(long *)(@esp+0xC) }; g"
# 64-bit
bp user32!NtUserSetWindowPos ".printf \"SetWindowPos( hWnd: %p, hWndInsertAfter: %p, X: %d, Y: %d, cx: %d, cy: %d, uFlags: %x )\\n\", @rcx, @rdx, @r8, @r9, poi(@rsp+0x20), poi(@rsp+0x28), poi(@rsp+0x30); g"
bp user32!NtUserShowWindow ".printf \"ShowWindow( hWnd: %p, nCmdShow: %d )\\n\", @rcx, @rdx; g"
bp user32!SetWindowLongW ".printf \"SetWindowLongW( hWnd: %p, nIndex: %d, dwNewLong: %p )\\n\", @rcx, @rdx, @r8; g"
bp user32!SetForegroundWindow ".printf \"SetForegroundWindow( hWnd: %p )\\n\", @rcx; g"
bp user32!NtUserSetParent ".printf \"SetParent( hWndChild: %p, hWndNewParent: %p )\\n\", @rcx, @rdx; g"
# 64-bit, but using dx
bp user32!NtUserSetWindowPos "dx new { function = \"SetWindowPos\", hWnd = (void *)@rcx, hWndInsertAfter = (void *)@rdx, X = (int)@r8, Y = (int)@r9, cx = *(int *)(@rsp+0x28), cy = *(int *)(@rsp+0x30), uFlags = *(unsigned int *)(@rsp+0x38) }; g"
bp user32!SetForegroundWindow "dx new { function = \"SetForegroundWindow\", hWnd = (void *)@rcx }; g"
bp user32!NtUserShowWindow "dx new { function = \"ShowWindow\", hWnd = (void *)@rcx, nCmdShow = (int)@rdx }; g"
bp user32!_imp_NtUserSetParent "dx new { function = \"SetParent\", hWndChild = (void *)@rcx, hWndNewParent = (void *)@rdx }; g"
bp user32!SetWindowLongW "dx new { function = \"SetWindowLongW\", hWnd = (void *)@rcx, nIndex = (int)@rdx, dwNewLong = (long)@r8 }; g"
# conditional breakpoints
bp user32!PeekMessageW "r $t1 = poi(@esp+4); bp /1 @$ra \".lastevent; dt (combase!tagMSG)@$t1; g\"; g"
bp user32!PeekMessageW ".lastevent; r $t1 = poi(@esp+4); r $t2 = poi(@esp+8); .printf \"PeekMessageW(%x, %x)\n\", @$t1, @$t2; ba e1 /1 @$ra \".if (poi(@$t1) == 0x40526) { .lastevent; dt (combase!tagMSG)@$t1; g } .else { g }\"; g"
bp user32!PeekMessageW "r $t1 = poi(@esp+4); ba e1 /1 @$ra \".if (poi(@$t1) == 0x7049c) { .lastevent; dt (combase!tagMSG)@$t1; g } .else { g }\"; g"
bp user32!SetWindowLongW ".lastevent; dps @esp L4; r $t0 = poi(@esp+c); .if ($t0 = 0) { g }"
bp user32!SetWindowLongW ".lastevent; dps @esp L4; r $t0 = poi(@esp+8); .if ($t0 = 0xffffffeb) { r @eip; } .else { g }"
bp user32!SetWindowLongW ".lastevent; dps @esp L4; .if (poi(@esp+8) = -2) { r @eip; } .else { g }"
```
When analyzing a TTD trace, it is quicker to list the function calls while extracting their parameters to anonymous objects, for example:
```cpp
dx -g @$cursession.TTD.Calls("user32!NtUserSetWindowPos").Select(c => new { HWND = c.Parameters[0], WClass = @$scriptContents.findWindow(c.Parameters[0]).className, X = c.Parameters[2], Y = c.Parameters[3], TimeStart = c.TimeStart, SystemTime = c.SystemTimeStart })
dx -g @$cursession.TTD.Calls("user32!SetParentStub").Select(c => new { Child = c.Parameters[0], ChildClass = @$scriptContents.findWindow(c.Parameters[0]).className, Parent = c.Parameters[1], ParentClass = @$scriptContents.findWindow(c.Parameters[1]).className, TimeStart = c.TimeStart, SystemTime = c.SystemTimeStart })
```
I also created a [**winapi-user32.ps1**](/assets/other/winapi-user32.ps1.txt) script, which decodes some of the window flag values to their text representation, for example:
```sh
# load script
. winapi-user32.ps1
# decode GWL_STYLE flag
Get-EnumFlagsFromMask -Enum ([GWL_STYLE]) -Mask 382664704
# WS_MAXIMIZEBOX
# WS_MINIMIZEBOX
# WS_THICKFRAME
# WS_SYSMENU
# WS_DLGFRAME
# WS_BORDER
# WS_CAPTION
# WS_CLIPCHILDREN
# WS_CLIPSIBLINGS
# WS_VISIBLE
# decode GWL_EXSTYLE flag
Get-EnumFlagsFromMask -Enum ([GWL_EXSTYLE]) -Mask 262400
# WS_EX_WINDOWEDGE
# WS_EX_APPWINDOW
Get-EnumFlagsFromMask ([SWP]) 20
# SWP_NOZORDER
# SWP_NOACTIVATE
```
{% endraw %}
================================================
FILE: guides/ebpf.md
================================================
---
layout: page
title: eBPF
date: 2025-12-22 08:00:00 +0200
---
{% raw %}
**Table of contents:**
<!-- MarkdownTOC -->
- [General information](#general-information)
- [bpftrace](#bpftrace)
- [Probe Metadata](#probe-metadata)
- [Language Syntax](#language-syntax)
- [CPU Sampling](#cpu-sampling)
- [Available functions](#available-functions)
- [My one-liners](#my-one-liners)
<!-- /MarkdownTOC -->
General information
-------------------
[Main project page](https://ebpf.io/)
To use eBPF you need to hold the following **required capabilities**: `CAP_BPF`, `CAP_PERFMON` (loading tracing programs), `CAP_NET_ADMIN` (loading network programs).
bpftrace
--------
### Probe Metadata
Information about available probes (instrumentation point for capturing event data) can be retrieved with the **-l** option, e.g.:
```shell
bpftrace -l 'tracepoint:syscalls:*execve*'
# tracepoint:syscalls:sys_enter_execve
# tracepoint:syscalls:sys_enter_execveat
# tracepoint:syscalls:sys_exit_execve
# tracepoint:syscalls:sys_exit_execveat
# and parameters
bpftrace -lv 'tracepoint:syscalls:sys_enter_execve*'
# tracepoint:syscalls:sys_enter_execve
# int __syscall_nr
# const char * filename
# const char *const * argv
# const char *const * envp
# tracepoint:syscalls:sys_enter_execveat
# int __syscall_nr
# int fd
# const char * filename
# const char *const * argv
# const char *const * envp
# int flags
```
### Language Syntax
In each event, we can reference one of the **[built-in variables](https://github.com/bpftrace/bpftrace/blob/master/man/adoc/bpftrace.adoc#builtins)**, including: `comm` (process name), `pid`, `tid`, or `args` (a special variable that allows us to access arguments of a given event, e.g., `args.filename` for `tracepoint:syscalls:sys_enter_openat`). Additionally, we can create so-called **scratch variables**, which will only be visible within a given probe:
```perl
BEGIN { let $n = (uint8)1; }
END { printf("%d", $n) } # error - $n is not available
```
Hashmaps, on the other hand, remain visible throughout the entire script execution:
```perl
BEGIN {
@myconf["only_stacks"] = (uint8)0;
}
END {
printf("Stats enabled: %d\n", @myconf["only_stats"]);
delete(@myconf, "only_stacks");
}
```
We also cannot change the type of either a variable or a value stored in a map, e.g.:
```perl
@myconf["pid"] = $# > 1 ? $2 : 0;
@myconf["pid"] = "test"; # error
```
The default numeric type is `uint64` and everything is cast to it, e.g.:
```perl
# pid will be stored as uint64 even if we cast to int32
BEGIN {
@myconf["pid"] = $# > 1 ? $2 : 0;
}
# WARNING: comparison of integers of different signs: 'int32' and 'uint64' can lead to undefined behavior
tracepoint:sched:sched_process_fork / @myconf["pid"] != 0 && args.parent_pid == @myconf["pid"] /
# OK:
tracepoint:sched:sched_process_fork / @myconf["pid"] != 0 && args.parent_pid == (int32)@myconf["pid"] /
```
At the beginning of a script, we can use **preprocessor directives**, but `#define` only works for constants (when I tried to add a function, I got a strange error).
In the preamble, we can also create our own types, but it seems we can only use them when working with C pointers. I couldn't initialize a variable of my type (`$t : my_struct = {}`). However, we can use tuples, which should often be sufficient. We reference tuple fields by their numeric index, e.g.:
```perl
tracepoint:syscalls:sys_enter_openat
{
@openat[tid] = (args.dfd, args.filename, args.mode);
}
tracepoint:syscalls:sys_exit_openat
{
$eventName = "file_openat";
PRINT_EVENT_COMMONS;
$data = @openat[tid];
printf("dfd: %d filename: '%s', mode: %x, ret: %d\n", $data.0, str($data.1), $data.2, args.ret);
delete(@openat[tid]);
}
```
`BEGIN` is one of the **[available probes](https://github.com/bpftrace/bpftrace/blob/master/man/adoc/bpftrace.adoc#probes)** that allows code execution at the start of a tracing session, e.g.:
```shell
bpftrace -e 'BEGIN { printf("hello world\n"); }'
# Attaching 1 probe...
# hello world
# ^C
```
If we prefix a variable name with `@`, we get a hashmap (without a name, we'll use the global hashmap). We can access its keys through square brackets. At the end of tracing, bpftrace outputs all used hashmaps, e.g.:
```perl
bpftrace -e 'tracepoint:syscalls:sys_enter_write { @[comm] = count(); }'
# Attaching 1 probe...
# ^C
#
# @[rtkit-daemon]: 1
# @[Worker Launcher]: 1
# ...
bpftrace -e 'tracepoint:syscalls:sys_enter_write { @write[comm] = count(); } tracepoint:syscalls:sys_enter_writev { @writev[comm] = count(); }'
# Attaching 2 probes...
# ^C
#
# @write[redshift-gtk]: 2
# @write[syncthing]: 2
# @write[redshift]: 2
# @writev[at-spi2-registr]: 2
# @writev[redshift]: 4
```
If we add `/ ... /` after a syscall name, we can place **a filter** between the slashes, e.g., `pid == 1234`, to display events only for the process with ID 1234, e.g.:
```perl
bpftrace -e 'tracepoint:syscalls:sys_enter_write / comm == "fish" / { @ = count(); }'
# Attaching 1 probe...
# ^C
#
# @: 415
```
We can also use ifs inside action code.
`count` is one of the **[map functions](https://github.com/bpftrace/bpftrace/blob/master/man/adoc/bpftrace.adoc#map-functions)** we can use to generate hashmaps. `hist`, `stats` and `avg` are other such functions.
### CPU Sampling
bpftrace [supports debuginfod symbols](https://github.com/iovisor/bcc/pull/3393/files) and this is awesome because, for example, ustack or kstack show real stacks. After collecting a trace, it can be converted to a flame graph using scripts from the [FlameGraph](https://github.com/brendangregg/FlameGraph) repository, e.g.:
```perl
bpftrace -o test-service.out -q -e 'profile:hz:99 / comm == "test-service" / { @[ustack()] = count(); }'
./stackcollapse-bpftrace.pl test-service.out > test-service.flame
./flamegraph.pl test-service.flame > test-service.flame.svg
```
### Available functions
In the **printf** function, the '-' character before the width means the text will be left-aligned, e.g.:
```perl
printf("|%-15s|\n", "TIME");
#|TIME |
printf("|%15s|\n", "TIME");
#| TIME|
```
### My one-liners
```perl
# openat with process information and collected stack
bpftrace -e 'tracepoint:syscalls:sys_enter_openat / strcontains(comm, "dump-") == 1 / { printf("%d:%s %d %s\n", pid, comm, args.dfd, str(args.filename)); print(ustack()); }'
```
{% endraw %}
================================================
FILE: guides/etw.md
================================================
---
layout: page
title: Event Tracing for Windows (ETW)
date: 2025-10-02 08:00:00 +0200
redirect_from:
- /guides/using-etw/
---
{% raw %}
**Table of contents:**
<!-- MarkdownTOC -->
- [General information](#general-information)
- [Tools](#tools)
- [Windows Performance Recorder \(WPR\)](#windows-performance-recorder-wpr)
- [Profiles](#profiles)
- [Starting and stopping the trace](#starting-and-stopping-the-trace)
- [Issues](#issues)
- [Windows Performance Analyzer \(WPA\)](#windows-performance-analyzer-wpa)
- [Installation](#installation)
- [Tips on analyzing events](#tips-on-analyzing-events)
- [Perfview](#perfview)
- [Installation](#installation_1)
- [Tips on recording events](#tips-on-recording-events)
- [Tips on analyzing events](#tips-on-analyzing-events_1)
- [Live view of events](#live-view-of-events)
- [Issues](#issues_1)
- [logman](#logman)
- [Querying providers installed in the system](#querying-providers-installed-in-the-system)
- [Starting and stopping the trace](#starting-and-stopping-the-trace_1)
- [wevtutil](#wevtutil)
- [tracerpt](#tracerpt)
- [xperf](#xperf)
- [TSS \(TroubleShootingScript toolset\)](#tss-troubleshootingscript-toolset)
- [MSO scripts \(PowerShell\)](#mso-scripts-powershell)
- [Event types](#event-types)
- [Autologger events](#autologger-events)
- [System boot events](#system-boot-events)
- [File events](#file-events)
- [Registry events](#registry-events)
- [WPP events](#wpp-events)
- [Libraries](#libraries)
- [ETW tools and libs \(including EtwEnumerator\)](#etw-tools-and-libs-including-etwenumerator)
- [TraceProcessing](#traceprocessing)
- [WPRContol](#wprcontol)
- [TraceEvent](#traceevent)
- [KrabsETW](#krabsetw)
- [Performance Logs and Alerts \(PLA\)](#performance-logs-and-alerts-pla)
- [System API](#system-api)
<!-- /MarkdownTOC -->
General information
-------------------
When loading **symbols**, the ETW tools and libraries use the **\_NT\_SYMBOLS\_PATH** environment variable to download (and cache) the PDB files and **\_NT\_SYMCACHE\_PATH** to store their preprocessed (cached) versions. An example machine configuration might look as follows:
```shell
setx /M _NT_SYMBOL_PATH "SRV*C:\symbols\dbg*https://msdl.microsoft.com/download/symbols"
setx /M _NT_SYMCACHE_PATH "C:\symcache"
```
On Windows 7 64-bit, to improve stack walking, disable paging of the drivers and kernel-mode system code:
```sh
reg add "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management" -v DisablePagingExecutive -d 0x1 -t REG\_DWORD -f
# or
wpr.exe -disablepagingexecutive`
```
For **manifest-based providers** set `MatchAnyKeywords` to `0x00` to receive all events. Otherwise you need to create a bitmask which will be or-ed with event keywords. Additionally when `MatchAllKeywords` is set, its value is used for events that passed the `MatchAnyKeywords` test and providers additional and filtering.
For **classic providers** set `MatchAnyKeywords` to `0xFFFFFFFF` to receive all events.
Up to 8 sessions may collect manifest-based provider events, but only 1 session may be created for a classic provider (when a new session is created the provider switches to the session).
When creating a session we may also specify the minimal severity level for collected events, where `1` is the critical level and `5` the verbose level (all events are logged).
Tools
-----
### Windows Performance Recorder (WPR)
#### Profiles
As its name suggests, WPR is a tool that records ETW traces and is available on all modern Windowses. It is straightforward to use and provides a big number of **ready-to-use tracing profiles**. We can list them with the `-profiles` command and show any profile details with the `profiledetails` command, for example:
```shell
# list available profiles with their short description
wpr.exe -profiles
# ...
# GeneralProfile First level triage
# CPU CPU usage
# DiskIO Disk I/O activity
# FileIO File I/O activity
# ...
# show profile details
wpr.exe -profiledetails CPU
# ...
# Profile : CPU.Verbose.Memory
#
# Collector Name : WPR_initiated_WprApp_WPR System Collector
# Buffer Size (KB) : 1024
# Number of Buffers : 3258
# Providers
# System Keywords
# CpuConfig
# CSwitch
# ...
# SampledProfile
# ThreadPriority
# System Stacks
# CSwitch
# ReadyThread
# SampledProfile
#
# Collector Name : WPR_initiated_WprApp_WPR Event Collector
# Buffer Size (KB) : 1024
# Number of Buffers : 20
# Providers
# b7a19fcd-15ba-41ba-a3d7-dc352d5f79ba: : 0xff
# e7ef96be-969f-414f-97d7-3ddb7b558ccc: 0x2000: 0xff
# Microsoft-JScript: 0x1: 0xff
# Microsoft-Windows-BrokerInfrastructure: 0x1: 0xff
# Microsoft-Windows-DotNETRuntime: 0x20098: 0x05
# ...
# Microsoft-Windows-Win32k: 0x80000: 0xff
```
Profiles often come in two versions: verbose and light, and we decide which one to use by appending "Verbose" or "Light" to the main profile name (if we do not specify the version, WPR defaults to "Verbose"), for example:
```sh
wpr.exe -profiledetails CPU.Light
```
The trace could be memory- or file- based, with memory-based being the default. We can switch to the file-based profile by using the `-filemode` option. If we can find a profile for our tracing scenario, we may build a custom one (WPR profile schema is documented [here](https://learn.microsoft.com/en-us/windows-hardware/test/wpt/recording-profile-xml-reference)). It is often easier to base it one of the existing profiles, which we may extract with the `-exportprofile` command, for example:
```sh
# export the memory-based CPU.Light profilek
wpr.exe -exportprofile CPU.Light C:\temp\CPU.light.wprp
# export the file-based CPU.Light profilek
wpr.exe -exportprofile CPU.Light C:\temp\CPU.light.wprp -filemode
```
Interestingly, in the XML file, profile names include also the tracing mode, so the memory-based profile will have name `CPU.Light.Memory`, as you can see in the example below:
```xml
<WindowsPerformanceRecorder Version="1.0">
<Profiles>
<!-- ... -->
<Profile Id="CPU.Light.Memory" Name="CPU" Description="RunningProfile:CPU.Light.Memory" LoggingMode="Memory" DetailLevel="Light">
<!-- or with the -filemode option -->
<Profile Id="CPU.Light.File" Name="CPU" Description="RunningProfile:CPU.Light.File" LoggingMode="File" DetailLevel="Light">
</Profiles>
</WindowsPerformanceRecorder>
```
An exteremly important parameter of the collector configuration are buffers. If we look into the exported profiles, we will find that the number of buffers differs depending on the mode which we use for tracing. Memory-based profiles will use a much higher number of buffers, for example:
```xml
<!-- CPU.Verbose.Memory -->
<SystemCollector Id="WPR_initiated_WprApp_WPR_System_Collector" Name="WPR_initiated_WprApp_WPR System Collector">
<BufferSize Value="1024" />
<Buffers Value="1023" />
</SystemCollector>
<!-- CPU.Verbose.File -->
<SystemCollector Id="WPR_initiated_WprApp_WPR_System_Collector" Name="WPR_initiated_WprApp_WPR System Collector">
<BufferSize Value="1024" />
<Buffers Value="20" />
</SystemCollector>
```
The number of buffers depends also on the amount of memory on the host. Because `BufferSize` specifies memory size in KB, the above space is quite large (1GB). In memory mode, we operate on circular in-memory buffers - the system adds new buffers when the previous buffers fill up. When it reaches the maximum, it begins to overwrite events in the oldest buffers. For a file-based traces, the number of buffers is much smaller, as we only need to ensure that we are not dropping events because the disk cannot keep up with the write operations.
Apart from keywords and levels, we may **[filter the trace and stack events](https://devblogs.microsoft.com/performance-diagnostics/filtering-events-using-wpr/)** by the event IDs (`EventFilters`, `StackFilters`). Filtering by process name is also possible, however, in my tests I found that the `ProcessExeFilter` works only for processes already running when we start the trace:
```xml
<EventProvider Id="DotNetRuntime" Name="e13c0d23-ccbc-4e12-931b-d9cc2eee27e4" ProcessExeFilter="filecopy.exe">
<Keywords>
<Keyword Value="0x60098" />
</Keywords>
</EventProvider>
<Profile Id="Wtrace.Verbose.Memory" Name="Wtrace" LoggingMode="Memory" DetailLevel="Verbose" Description="wtrace trace in memory profile">
<Collectors>
<EventCollectorId Value="wtrace-user">
<EventProviders>
<EventProviderId Value="DotNetRuntime" />
</EventProviders>
</EventCollectorId>
</Collectors>
</Profile>
```
Working with WPR profiles is described in details in a great series of posts on [Microsoft's Performance and Diagnostics blog](https://devblogs.microsoft.com/performance-diagnostics/) and I highly recommend reading them:
- [WPR Start and Stop Commands](https://devblogs.microsoft.com/performance-diagnostics/wpr-start-and-stop-commands/)
- [Authoring custom profiles – Part 1](https://devblogs.microsoft.com/performance-diagnostics/authoring-custom-profiles-part-1/)
- [Authoring Custom Profiles – Part 2](https://devblogs.microsoft.com/performance-diagnostics/authoring-custom-profiles-part-2/)
- [Authoring Custom Profiles – Part 3](https://devblogs.microsoft.com/performance-diagnostics/authoring-custom-profile-part3/)
I also created **an [EtwMetadata.ps1](/assets/other/EtwMetadata.ps1.txt) script that you may use to decode the wprp files**. For example:
```sh
wpr.exe -exportprofile CPU.Light C:\temp\CPU.light.wprp
curl.exe -o C:\temp\EtwMetadata.ps1 https://wtrace.net/assets/other/EtwMetadata.ps1.txt
. C:\temp\EtwMetadata.ps1
# Initializing ETW providers metadata...
Get-EtwProvidersFromWprProfile C:\temp\CPU.light.wprp
# WARNING: No metadata found for provider 'b7a19fcd-15ba-41ba-a3d7-dc352d5f79ba'
# WARNING: No metadata found for provider 'e7ef96be-969f-414f-97d7-3ddb7b558ccc'
# Id Name Keywords
# -- ---- --------
# 36b6f488-aad7-48c2-afe3-d4ec2c8b46fa Microsoft-Windows-Performance-Recorder-Control @{Name=PerfStatus; Value=65536}
# b675ec37-bdb6-4648-bc92-f3fdc74d3ca2 Microsoft-Windows-Kernel-EventTracing @{Name=ETW_KEYWORD_LOST_EVENT; Val…
# 83ed54f0-4d48-4e45-b16e-726ffd1fa4af Microsoft-Windows-Networking-Correlation {@{Name=ActivityTransfer; Value=1}…
# d8975f88-7ddb-4ed0-91bf-3adf48c48e0c Microsoft-Windows-RPCSS {@{Name=EpmapDebug; Value=256}, @{…
# 6ad52b32-d609-4be9-ae07-ce8dae937e39 Microsoft-Windows-RPC
# d49918cf-9489-4bf1-9d7b-014d864cf71f Microsoft-Windows-ProcessStateManager {@{Name=StateChange; Value=1}, @{N…
# e6835967-e0d2-41fb-bcec-58387404e25a Microsoft-Windows-BrokerInfrastructure @{Name=BackgroundTask; Value=1}
```
#### Starting and stopping the trace
After picking a profile or profiles that we want to use, we can **start a tracing session** with the `-start` command. Some examples:
```sh
# starts verbose CPU profile
wpr.exe -start CPU.verbose
# same as above
wpr.exe -start CPU
# starts light CPU profile
wpr.exe -start CPU.light
# multiple profiles start
wpr.exe -start CPU -start VirtualAllocation -start Network
# starts a custom WPRTest.Verbose profile defined in the C:\temp\CustomProfile.wprp file
wpr.exe -start "C:\temp\CustomProfile.wprp!WPRTest" -filemode
# starts a custom WPRTest.Light profile defined in the C:\temp\CustomProfile.wprp file
wpr.exe -start "C:\temp\CustomProfile.wprp!WPRTest.Light"
```
There could be only one WPR trace running in the system and we can check its status using the `-status` command:
```sh
wpr -status
# Microsoft Windows Performance Recorder Version 10.0.26100 (CoreSystem)
# Copyright (c) 2024 Microsoft Corporation. All rights reserved.
#
# WPR recording is in progress...
#
# Time since start : 00:00:01
# Dropped event : 0
# Logging mode : File
```
To **terminate the trace** we may use either the `-stop` or the `-cancel` command:
```shell
# stopping the trace and saving it to a file with an optional description
wpr.exe -stop "C:\temp\testapp-fail.etl" "Abnormal termination of testapp.exe"
# cancelling the trace (no trace files will be created)
wpr.exe -cancel
```
#### Issues
##### Error 0x80010106 (RPC_E_CHANGED_MODE)
If it happens when you run the `-stop` command, use wpr.exe from Windows SDK, build 1950 or later.
##### Error 0xc5580612
If you are using `ProcessExeFilter` in your profile, this error may indicate that a process with a given name is not running when the trace starts (it is thrown by `WindowsPerformanceRecorderControl!WindowsPerformanceRecorder::CControlManager::VerifyAllProvidersEnabled`):
```
An Event session cannot be started without any providers.
Profile Id: Wtrace.Verbose.File
Error code: 0xc5580612
An Event session cannot be started without any providers.
```
### Windows Performance Analyzer (WPA)
#### Installation
**Windows Performance Analyzer (wpa.exe)**, may be installed from [Microsoft Store](https://apps.microsoft.com/store/detail/windows-performance-analyzer-preview/9N58QRW40DFW?hl=en-sh&gl=sh) (recommended) or as part of the **Windows Performance Toolkit**, included in the [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/).
#### Tips on analyzing events
In **CPU Wait analysis**, each row marks a moment, when a thread received CPU time ([MS docs](https://learn.microsoft.com/en-us/windows-hardware/test/wpt/cpu-analysis#cpu-usage-precise-graph)) after, for example, waiting on an event object. The `Readying Thread` is the thread that woke up the `New Thread`. And the `Old Thread` is the thread which gave place on a CPU to the `New Thread`. The diagram below from Microsoft documentation nicely explain those terms:
Here is an example view of my test GUI app when I call the `Sleep` function after pressing a button:
As you can see, the `Wait` column shows the time spent on waiting, while the UI view shows the time when the application was unresponsive.
WPA allows us to **group the call stacks** by tags. The default stacktag list can be found in the `c:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\Catalog\default.stacktags` file.
We may also **extend WPA with our own plugins**. The [SDK repository](https://github.com/microsoft/microsoft-performance-toolkit-sdk/) contains sample extensions. [Wpa.Demystifier](https://github.com/Zhentar/Wpa.Demystifier/tree/master) is another interesting extension to check.
### Perfview
#### Installation
Could be downloaded from [its release page](https://github.com/microsoft/perfview/releases) or installed with winget:
```sh
winget install --id Microsoft.PerfView
```
#### Tips on recording events
Most often you will use the Collect dialog, but it is also possible to use PerfView from a command line. An example command collecting traces into a 500MB file (in circular mode) may look as follows:
```sh
perfview -AcceptEULA -ThreadTime -CircularMB:500 -Circular:1 -LogFile:perf.output -Merge:TRUE -Zip:TRUE -noView collect
```
A new console window will open with the following text:
```
Pre V4.0 .NET Rundown enabled, Type 'D' to disable and speed up .NET Rundown.
Do NOT close this console window. It will leave collection on!
Type S to stop collection, 'A' will abort. (Also consider /MaxCollectSec:N)
Type 'S' when you are done with tracing and wait (DO NOT CLOSE THE WINDOW) till you see `Press enter to close window`. Then copy the files: PerfViewData.etl.zip and perf.output to the machine when you will perform analysis.
```
If you are also interested in the network traces append the `-NetMonCapture` option. This will generate an additional PerfViewData_netmon.cab file.
If we use the EventSource provider and want to collect the call stacks along with the events, we need to append `@StacksEnabled=true` to the provider name, for example: `*EFTrace:@StacksEnabled=true`.
#### Tips on analyzing events
Select a **time range** and press `Alt+R` to set it for the grid. We may also copy a range, paste it in the Start box and then press Enter to apply it (PerfView should fill the End box).
The table below contains grouping patterns I use for various analysis targets
Name | Pattern
-------- | --------
Just my code with folded threads | `[My app + folded threads] \Temporary ASP.NET Files\->;!dynamicClass.S->;!=>OTHER;Thread->AllThreads` |
Just my code with folded threads (ASP.NET view) | `[My app + folded threads and ASP.NET requests] Thread -> AllThreads;Request ID * URL: {*}-> URL $1;\Temporary ASP.NET Files\->;!dynamicClass.S->;!=>OTHER`
Just my code with folded threads (Server requests view) | `[My app + folded threads and requests] Thread -> AllThreads;ASP.NET Request: * URL: {*}-> URL $1;\Temporary ASP.NET Files\->;!dynamicClass.S->;!=>OTHER`
Group requests | `^Request ID->ALL Requests`
Group requests by URL | `Request ID * URL:{*}->$1`
Group async calls (by Christophe Nasarre) | `{%}!{%}+<>c__DisplayClass*+<<{%}>b__*>d.MoveNext()->($1) $2 async $3`
When exporting to **Excel**, the data coming from PerfView often does not have valid formatting and contains some strange characters at the beginning or at the end, for example:
```
0000 A0 A0 32 32 34 224
```
We may clean up those values by using the **SUBSTITUTE** function, for example:
```
=SUBSTITUTE(A1,LEFT(A1,1),"")
=SUBSTITUTE(A1,RIGHT(A1,1),"")
```
And later do the usual Copy, Paste as Values operation. Alternatively, we may copy the values column by column. In that case, PerfView won't insert those special characters.
If we want to open a trace created by PerfView in **WPA**, we need to first convert it, for example:
```sh
perfview /wpr unzip test.etl.zip
# The above command should create two files (.etl and .etl.ngenpdb)
# and we can open wpa
wpa test.etl
```
#### Live view of events
The `Listen` user command enables a live view dump of events in the PerfView log:
```sh
PerfView.exe UserCommand Listen Microsoft-JScript:0x7:Verbose
# inspired by Konrad Kokosa's tweet
PerfView.exe UserCommand Listen Microsoft-Windows-DotNETRuntime:0x1:Verbose:@EventIDsToEnable="1 2"
```
#### Issues
##### Error 0x800700B7 (ERROR_ALREADY_EXISTS)
```
[Kernel Log: C:\tools\PerfViewData.kernel.etl]
Kernel keywords enabled: Default
Aborting tracing for sessions 'NT Kernel Logger' and 'PerfViewSession'.
Insuring .NET Allocation profiler not installed.
Completed: Collecting data C:\tools\PerfViewData.etl (Elapsed Time: 0,858 sec)
Exception Occured: System.Runtime.InteropServices.COMException (0x800700B7): Cannot create a file when that file already exists. (Exception from HRESULT: 0x800700B7)
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at Microsoft.Diagnostics.Tracing.Session.TraceEventSession.EnableKernelProvider(Keywords flags, Keywords stackCapture)
at PerfView.CommandProcessor.Start(CommandLineArgs parsedArgs)
at PerfView.CommandProcessor.Collect(CommandLineArgs parsedArgs)
at PerfView.MainWindow.c__DisplayClass9.b__7()
at PerfView.StatusBar.c__DisplayClass8.b__6(Object param0)
An exceptional condition occurred, see log for details.
```
If you receive such error, make sure that no kernel log is running with `perfview listsessions` and eventually kill it with `perfview abort`.
### logman
Nowadays, logman will not be our first choice tool to collect ETW trace, but the best thing about it is that it is a built-in tool and has been available in Windows for many years already, so might be the only option if you need to work on a legacy Windows system.
#### Querying providers installed in the system
Logman is great for querying ETW providers installed in the system or activated in a given process:
```sh
# list all providers in the system
logman query providers
# show details about the ".NET Common Language Runtime" provider
logman query providers ".NET Common Language Runtime"
# list providers active in a process with ID 808
logman query providers -pid 808
```
#### Starting and stopping the trace
The following commands start and stop a tracing session that is using one provider:
```sh
logman start mysession -p {9744AD71-6D44-4462-8694-46BD49FC7C0C} -o "c:\temp\test.etl" -ets & timeout -1 & logman stop mysession -ets
```
For the provider options you may additionally specify the keywords (flags) and levels that will be logged: `-p provider [flags [level]]`
You may also use a file with a list of providers:
```sh
logman start mysession -pf providers.guids -o c:\temp\test.etl -ets & timeout -1 & logman stop mysession -ets
```
And the `providers.guids` file content is built of lines following the format: `{guid} [flags] [level] [provider name]` (flags, level, and provider name are optional), for example:
```
{AFF081FE-0247-4275-9C4E-021F3DC1DA35} 0xf 5 ASP.NET Events
{3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83} 0x1ffe 5 IIS: WWW Server
```
If you want to record events from the **kernel provider** you need to name the session: `NT Kernel Logger`, for example:
```sh
logman start "NT Kernel Logger" -p "Windows Kernel Trace" "(process,thread,file,fileio,net)" -o c:\kernel.etl -ets & timeout -1 & logman stop "NT Kernel Logger" -ets
```
To see the available kernel provider keywords, run:
```sh
logman query providers "Windows Kernel Trace"
# Provider GUID
# -------------------------------------------------------------------------------
# Windows Kernel Trace {9E814AAD-3204-11D2-9A82-006008A86939}
#
# Value Keyword Description
# -------------------------------------------------------------------------------
# 0x0000000000000001 process Process creations/deletions
# 0x0000000000000002 thread Thread creations/deletions
# ...
```
Additionally, we may change the way how events are saved to the file using the `-mode` parameter. For example, to use a circular file with maximum size of 200MB, we can run the following command:
```sh
logman start "NT Kernel Logger" -p "Windows Kernel Trace" "(process,thread,img)" -o C:\ntlm-kernel.etl -mode circular -max 200 -ets
```
### wevtutil
Wevtutil is a built-in tool that allows us to manage **manifest-based providers (publishers)** installed in our system. Example usages:
```sh
# list all installed publishers
wevtutil ep
# find MSMQ publishers
wevtutil ep | findstr /i msmq
# extract details about a Microsoft-Windows-MSMQ publisher
wevtutil gp Microsoft-Windows-MSMQ /ge /gm /f:xml
```
### tracerpt
Tracerpt is another built-in tool. It may collect ETW traces, but I usually use it only to convert etl files from binary to text format. Example commands:
```sh
# convert etl file to evtx
tracerpt -of EVTX test.etl -o test.evtx -summary test-summary.xml
# dump events to an XML file
tracerpt test.etl -o test.xml -summary test-summary.xml
# dump events to a HTML file
tracerpt.exe '.\NT Kernel Logger.etl' -o -report -f html
```
### xperf
For a long time xperf was the best tool to collect ETW traces, providing ways to configure many aspects of the tracing sessions. It is now considered legacy (with [wpr](#windows-performance-recorder-wpr) being its replacement), but many people still find its command line syntax eaier to use than WPR profiles. Here are some usage examples:
```sh
# list available Kernel Flags
xperf -providers KF
# PROC_THREAD : Process and Thread create/delete
# LOADER : Kernel and user mode Image Load/Unload events
# PROFILE : CPU Sample profile
# CSWITCH : Context Switch
# ...
# list available Kernel Groups
xperf -providers KG
# Base : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+PROFILE+MEMINFO+MEMINFO_WS
# Diag : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PERF_COUNTER+COMPACT_CSWITCH
# DiagEasy : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PERF_COUNTER
# ...
# list installed providers
xperf -providers I
# 0063715b-eeda-4007-9429-ad526f62696e : Microsoft-Windows-Services
# 0075e1ab-e1d1-5d1f-35f5-da36fb4f41b1 : Microsoft-Windows-Network-ExecutionContext
# 00b7e1df-b469-4c69-9c41-53a6576e3dad : Microsoft-Windows-Security-IdentityStore
# 01090065-b467-4503-9b28-533766761087 : Microsoft-Windows-ParentalControls
# ...
# start the kernel trace, enabling flags defined in the DiagEasy group
xperf -on DiagEasy
# stop the kernel trace
xperf -stop -d "c:\temp\DiagEasy.etl"
# start the kernel with some additional settings and wait for the user to stop it
xperf -on Latency -stackwalk Profile -buffersize 2048 -MaxFile 1024 -FileMode Circular && timeout -1 && xperf stop -d "C:\highCPUUsage.etl"
# in user-mode tracing you may still use kernel flags and groups but for each user-trace provider
# you need to add some additional parameters: -on (GUID|KnownProviderName)[:Flags[:Level[:0xnnnnnnnn|'stack|[,]sid|[,]tsid']]]
xperf -start ClrRundownSession -on ClrAll:0x118:5+a669021c-c450-4609-a035-5af59af4df18:0x118:5 -f clr_DCend.etl -buffersize 128 -minbuffers 256 -maxbuffers 512
timeout /t 15
xperf -stop ClrSession ClrRundownSession -stop -d cpu_clr.etl
# dump collected events to a text file
xperf -i test.etl -o test.csv
```
Chad Schultz published [many xperf scripts](https://github.com/itoleck/WindowsPerformance/tree/main/ETW/Tools/WPT/Xperf/CaptureScripts) in the [WindowsPerformance repository](https://github.com/itoleck/WindowsPerformance), so check them out if you are interested in using xperf.
### TSS (TroubleShootingScript toolset)
TSS contains tons of various scripts and ETW is only a part of it. TSS official documentation is [here](https://learn.microsoft.com/en-us/troubleshoot/windows-client/windows-tss/introduction-to-troubleshootingscript-toolset-tss) and we can download the package from <https://aka.ms/getTSS>.
Here is an example PowerShell script to install and run the main script:
```shell
powershell.exe -NoProfile -ExecutionPolicy RemoteSigned -Command "Invoke-WebRequest -Uri https://aka.ms/getTSS -OutFile $env:TEMP\TSS.zip; Unblock-File $env:TEMP\TSS.zip; Expand-Archive -Force -LiteralPath $env:TEMP\TSS.zip -DestinationPath C:\TSS; Remove-Item $env:TEMP\TSS.zip; C:\TSS\TSS.ps1 -ListSupportedTrace"
```
TSS defined many **troubleshooting scenarios** with precompiled parameters:
```shell
C:\tSS\TSS.ps1 -ListSupportedScenarioTrace
# ...
# NET_General - collects CommonTask NET, NetshScenario InternetClient_dbg, Procmon, PSR, Video, SDP NET, xray, CollectComponentLog
# ...
```
where:
- `CommonTask` are commands run before and after the scenario (only `NET` in this case)
- `NetshScenario` is the selected netsh scenario (`InternetClient_dbg`)
- `Procmon` will start procmon
- `PSR` will run step recorder
- `Video` will record a video of what the user is doing
- `SDP` (Support Diagnostic Package) and `NET` enable `General`, `SMB`, and `NET` counters
- `xray` runs xray scripts to discover existing problems
- `CollectComponentLog` collects logs of commands run in a given scenario
To start a scenario, we run:
```shell
C:\TSS\TSS.ps1 -Scenario NET_General
```
We may also manually "compose" the TSS command. A nice GUI tool for this purpose is `.\TSSGUI.ps1` (start it from the TSS folder). We may also list available TSS features:
```shell
C:\TSS\TSS.ps1 -ListSupportedCommands
C:\TSS\TSS.ps1 -ListSupportedControls
C:\TSS\TSS.ps1 -ListSupportedDiag
C:\TSS\TSS.ps1 -ListSupportedLog
C:\TSS\TSS.ps1 -ListSupportedNetshScenario
C:\TSS\TSS.ps1 -ListSupportedNoOptions
C:\TSS\TSS.ps1 -ListSupportedPerfCounters
C:\TSS\TSS.ps1 -ListSupportedScenarioTrace
C:\TSS\TSS.ps1 -ListSupportedSDP
C:\TSS\TSS.ps1 -ListSupportedSetOptions
C:\TSS\TSS.ps1 -ListSupportedTrace
C:\TSS\TSS.ps1 -ListSupportedWPRScenario
C:\TSS\TSS.ps1 -ListSupportedXperfProfile
```
Example commands to check which ETW providers the `NET_COM` component is using:
```shell
.\TSS.ps1 -ListSupportedTrace | select-string "_COM"
# [Component] -NET_COM COM/DCOM/WinRT/PRC component tracing. -EnableCOMDebug will enable further debug logging
# [Component] -UEX_COM COM/DCOM/WinRT/PRC component ETW tracing. -EnableCOMDebug will enable further debug logging
# Usage:
# .\TSS.ps1 -<ComponentName> -<ComponentName>
# Example: .\TSS.ps1 -UEX_FSLogix -UEX_Logon
.\TSS -ListETWProviders NeT_COM
# List of 20 Provider GUIDs (Flags/Level) for ComponentName: NET_COM
# ==========================================================
# {9474a749-a98d-4f52-9f45-5b20247e4f01}
# {bda92ae8-9f11-4d49-ba1d-a4c2abca692e}
# ...
```
The TSS commands create raports in the `C:\MS_DATA` folder.
To collect the trace in the background we may use the `-StartNoWait` option and `-Stop` to stop the trace.
If we add the `-StartAutoLogger` option, our trace will start when the system boots. We stop by calling `TSS.ps1 -Stop`, as usual.
Example commands:
```shell
# starting WPR using TSS
C:\TSS\TSS.ps1 -WPR CPU -WPROptions "-start Dotnet -start DesktopComposition"
# Starting time travel debugging session using TSS
# 1234 is the process PID (we may use process name as well, for example winver.exe)
C:\TSS\TSS.ps1 -AcceptEula -TTD 1234
```
### MSO scripts (PowerShell)
[MSO-Scripts repository](https://github.com/microsoft/MSO-Scripts) hosts many interesting PowerShell scripts for working with ETW traces.
Event types
-----------
### Autologger events
Autologger ETW session collects events appearing after the system start. It can be enabled with wpr:
```sh
wpr.exe -boottrace -addboot FileIO
```
Additional information:
- [Autologger session](https://learn.microsoft.com/en-us/windows/win32/etw/configuring-and-starting-an-autologger-session)
- [Autologger with WPR](https://devblogs.microsoft.com/performance-diagnostics/setting-up-an-autologger-with-wpr/)
### System boot events
To collect general profile traces use:
```sh
wpr.exe -start generalprofile -onoffscenario boot -numiterations 1
```
### File events
Described in [a post on my blog](https://lowleveldesign.org/2020/08/15/fixing-empty-paths-in-fileio-events-etw/).
### Registry events
Described in [a post on my blog](https://lowleveldesign.org/2020/08/20/monitoring-registry-activity-with-etw/).
### WPP events
WPP events are legacy events, for which we need TMF files to decode their payload. TMF may be available as standalone files or they might be embedded into PDB files. For the latter case, we may extract them using **tracepdb.exe**, for example:
```sh
tracepdb.exe -f .\combase.pdb -p .\tmfs
```
TMF data is stored as a binary block in the PDB file:
```
0D9:46A0 BA 00 19 10 20 52 0A 00 01 00 06 00 54 4D 46 3A º... R......TMF:
0D9:46B0 00 64 61 66 38 39 65 63 31 2D 64 66 66 32 2D 33 .daf89ec1-dff2-3
0D9:46C0 30 35 35 2D 36 30 61 62 2D 36 33 64 34 63 31 31 055-60ab-63d4c11
0D9:46D0 62 33 64 39 63 20 4F 4C 45 43 4F 4D 20 2F 2F 20 b3d9c OLECOM //
0D9:46E0 53 52 43 3D 63 6F 6D 74 72 61 63 65 77 6F 72 6B SRC=comtracework
0D9:46F0 65 72 2E 63 78 78 20 4D 4A 3D 20 4D 4E 3D 00 23 er.cxx MJ= MN=.#
0D9:4700 74 79 70 65 76 20 63 6F 6D 74 72 61 63 65 77 6F typev comtracewo
0D9:4710 72 6B 65 72 5F 63 78 78 31 38 36 20 31 31 20 22 rker_cxx186 11 "
0D9:4720 25 30 25 31 30 21 73 21 22 20 2F 2F 20 20 20 4C %0%10!s!" // L
0D9:4730 45 56 45 4C 3D 57 41 52 4E 49 4E 47 00 7B 00 6D EVEL=WARNING.{.m
0D9:4740 65 73 73 61 67 65 2C 20 49 74 65 6D 57 53 74 72 essage, ItemWStr
0D9:4750 69 6E 67 20 2D 2D 20 31 30 00 7D 00 BA 00 19 10 ing -- 10.}.º...
```
The GUID at the beginning of the block defines the provider ID and may appear multiple times in the PDB file. Tracepdb uses this ID as the name of the generated TMF file. When decoding WPP events, if we do not configure the `TDH_CONTEXT_WPP_TMFSEARCHPATH`, Tdh functions will look for TMF files in the path specified in the [TRACE_FORMAT_SEARCH_PATH environment variable](https://learn.microsoft.com/en-us/windows/win32/api/tdh/ne-tdh-tdh_context_type). **WPA** has a special view for WPP events and can load the TMF manifests from symbol files, so **remember to first load the symbols**.
Libraries
---------
This section lists some of the ETW libraries I used with my notes about them. It is not meant to be a comprehensive documentation of those libraries, but rather a list of tips and tricks.
### ETW tools and libs (including EtwEnumerator)
[Source code](https://github.com/microsoft/ETW)
This C++ library contains code to parse ETW events. The sample EtwEnumerator CLI tool formats events from a binary etl file to their text representation.
To build the library run:
```shell
cd EtwEnumerator
cmake -B bin .
cmake --build bin
```
The `EtwEnumerator` instance stores information about the currently analyzed event in an efficient way, caching metadata for future processing of similar events. Please check the [README](https://github.com/microsoft/ETW/tree/main/EtwEnumerator). Below is an example C# code that formats event to a JSON string in the [ETW callback function](https://learn.microsoft.com/en-us/windows/win32/api/evntrace/nc-evntrace-pevent_record_callback):
```cs
EtwStringViewZ etwString;
fixed (char* formatPtr = "[%9]%8.%3::%4 [%1]")
{
if (!ee->FormatCurrentEvent((ushort*)formatPtr, EtwJsonSuffixFlags.EtwJsonSuffixFlags_Default, &etwString))
{
Trace.WriteLine("ERROR");
return;
}
}
var s = new string((char*)etwString.Data, 0, (int)etwString.DataLength);
writer.TryWrite(new MessageEvent(s));
```
### TraceProcessing
[Documentation](https://learn.microsoft.com/en-us/windows/apps/trace-processing/) | [Code samples](https://github.com/microsoft/eventtracing-processing-samples)
TraceProcessing library **categorized the events and splits them between Trace Processor**. Before processing the trace, we mark Trace Processors that we want to active, and we may query the events they processed after the analysis finishes, for example:
```cs
using var trace = TraceProcessor.Create(traceFilePath);
var pendingProcesses = trace.UseProcesses();
var pendingFileIO = trace.UseFileIOData();
trace.Process();
var filecopyProcess = pendingProcesses.Result.Processes.Where(p => p.ImageName == "filecopy.exe").First();
var fev = pendingFileIO.Result.CreateFileObjectActivity.First(f => f.IssuingProcess.Id == filecopyProcess.Id
&& f.FileName == "sampling-2-1.etl");
Console.WriteLine($"Create file event: {fev.Path} ({fev.FileObject})");
```
The above code uses the buffered mode of opening a trace file, in which all processed events land in memory (we may notice that the application memory consumption will be really high for bigger traces). Therefore, for bigger traces we may also use [the streaming mode](https://learn.microsoft.com/en-us/windows/apps/trace-processing/streaming), but not all event types support it. An example session using streaming mode might be coded as follows:
```cs
using var trace = TraceProcessor.Create(traceFilePath);
var pendingProcesses = trace.UseProcesses();
int filecopyProcessId = 0;
long eventCount = 0;
long filecopyEventCount = 0;
// ConsumerSchedule defines when our parser will be called, for example, we may choose
// SecondPass when buffered processors will be available
trace.UseStreaming().UseUnparsedEvents(ConsumerSchedule.Default, context =>
{
eventCount++;
});
trace.UseStreaming().UseUnparsedEvents(ConsumerSchedule.SecondPass, context =>
{
if (filecopyProcessId == 0)
{
filecopyProcessId = pendingProcesses.Result.Processes.Where(p => p.ImageName == "filecopy.exe").First().Id;
}
if (context.Event.ProcessId == filecopyProcessId)
{
filecopyEventCount++;
}
});
trace.Process();
return (filecopyEventCount, eventCount);
```
In my tests, I discovered that **GenericEvents** processor is not very reliable as I could not find some of the events (for example, FileIo), visible in other tools, but maybe I was doing something wrong :)
### WPRContol
WPRControl is the COM object used by, for example, wpr.exe. Its API is [well-documented](https://learn.microsoft.com/en-us/windows-hardware/test/wpt/wprcontrol-api-reference), with `KernelTraceControl.h` and `WindowsPerformanceRecordedControl.h` headers and IDLs available for our usage.
### TraceEvent
[Source code](https://github.com/microsoft/perfview/tree/main/src/TraceEvent) | [Documentation](https://github.com/microsoft/perfview/tree/main/documentation)
TraceEvent is a huge library which is the tracing engine that PerfView uses for collecting and processing events.
When iterating through collected events, remember to clone the events you need for future processing as the current `TraceEvent` instance is in-memory replaced by the next analyzed event. For example the `requestStartEvent` and `requestStopEvent` in the code below will contain invalid data at the end of the loop (we should be calling `ev.Clone()` to save the event):
```cs
TraceEvent? requestStartEvent = null, requestStopEvent = null;
foreach (var ev in traceLog.Events.Where(ev => ev.ProviderGuid == aspNetProviderId))
{
if (ev.ActivityID == activityIdGuid)
{
if (ev.ID == (TraceEventID)2) // Request/Start
{
requestStartEvent = ev;
}
if (ev.ID == (TraceEventID)3) // Request/Stop
{
requestStopEvent = ev;
}
}
}
// requestStartEvent i requestStopEvent zawierają błędne dane, ponieważ obiekt, którego wewnętrznie używają ma nadpisane dane przez późniejsze eventy
```
If you are interested how TraceEvent library processes the ETW events, the good place to start is the `ETWTraceEventSource.RawDispatchClassic` event callback function. It uses `TraceEvent.Lookup` to create the final instance of the `TraceEvent` class.
### KrabsETW
[Source code](https://github.com/microsoft/krabsetw)
KrabsETW is used by the Office 365 Security team. An example code to start a live session looks as follows:
```cs
using Microsoft.O365.Security.ETW;
using Microsoft.O365.Security.ETW.Kernel;
using var trace = new KernelTrace("krabsetw-lab");
var processProvider = new ProcessProvider();
processProvider.OnEvent += (record) =>
{
if (record.Opcode == 0x01)
{
var image = record.GetAnsiString("ImageFileName", "Unknown");
var pid = record.GetUInt32("ProcessId", 0);
Console.WriteLine($"{image} started with PID {pid}");
}
};
trace.Enable(processProvider);
Console.CancelKeyPress += (sender, ev) =>
{
ev.Cancel = true;
trace.Stop();
};
trace.Start();
```
The KrabsETW is implemented in C++ CLI which complicates the deployment. Firstly, I needed to add `<RuntimeIdentifier>win-x64</RuntimeIdentifier>` to my csproj file to fix a problem with missing `Ijwhost.dll` library. However, it still produced errors when trimming and the application was failing:
```sh
dotnet publish -c release -r win-x64 -p:PublishSingleFile=true -p:PublishTrimmed=true --self-contained -p:IncludeNativeLibrariesForSelfExtract=true
# MSBuild version 17.6.8+c70978d4d for .NET
# Determining projects to restore...
# All projects are up-to-date for restore.
# krabsetw-lab -> C:\code\krabsetw-lab\bin\release\net7.0-windows\win-x64\krabsetw-lab.dl
# l
# Optimizing assemblies for size. This process might take a while.
# C:\Users\me\.nuget\packages\microsoft.o365.security.native.etw\4.3.1\lib\net6.0\Microsoft.O365.Security.Native.ETW.dll
# : warning IL2104: Assembly 'Microsoft.O365.Security.Native.ETW' produced trim warnings. For more information see https:
# //aka.ms/dotnet-illink/libraries [C:\code\krabsetw-lab\krabsetw-lab.csproj]
# krabsetw-lab -> C:\code\krabsetw-lab\bin\release\net7.0-windows\win-x64\publish\
```
```sh
krabsetw-lab.exe
# Unhandled exception. System.BadImageFormatException:
# File name: 'C:\code\krabsetw-lab\bin\release\net7.0-windows\win-x64\publish\Microsoft.O365.Security.Native.ETW.dll'
# at Program.<Main>$(String[] args)
```
When processing events, KrabsETW uses `schema_locator` to cache and decode payload of a given event:
```cpp
struct schema_key
{
guid provider;
uint16_t id;
uint8_t opcode;
uint8_t version;
uint8_t level;
// ...
}
inline const PTRACE_EVENT_INFO schema_locator::get_event_schema(const EVENT_RECORD &record) const
{
// check the cache
auto key = schema_key(record);
auto& buffer = cache_[key];
if (!buffer) {
auto temp = get_event_schema_from_tdh(record);
buffer.swap(temp);
}
return (PTRACE_EVENT_INFO)(buffer.get());
}
```
### Performance Logs and Alerts (PLA)
[Documentation](https://learn.microsoft.com/en-us/previous-versions/windows/desktop/pla/pla-portal)
PLA is a COM library used by logman to provide trace collection options. The library registration can be located in the registry:
```
Computer\HKEY_CLASSES_ROOT\CLSID\{03837513-098B-11D8-9414-505054503030}
```
The main DLLs are **pla.dll** and **plasrv.exe**.
For example, the `ITraceDataProviderCollection::GetTraceDataProvidersByProcess` method, responsible for querying providers in a process, calls `TraceSession::LoadGuidArray`, which then uses `EnumerateTraceGuidsEx`.
### System API
[Documentation](https://learn.microsoft.com/en-us/windows/win32/api/_etw/)
Low-level API to collect and analyze traces - all above libraries use these functions.
{% endraw %}
================================================
FILE: guides/gdb.md
================================================
---
layout: page
title: GDB usage guide
date: 2025-05-27 08:00:00 +0200
---
{% raw %}
**Table of contents:**
<!-- MarkdownTOC -->
- [Configuration](#configuration)
- [.gdbinit](#gdbinit)
- [ptrace capability](#ptrace-capability)
- [TUI](#tui)
- [Symbols](#symbols)
- [Searching for symbols and addresses](#searching-for-symbols-and-addresses)
- [Searching for source code](#searching-for-source-code)
- [Debugging child processes](#debugging-child-processes)
- [Execution Control](#execution-control)
- [Process startup](#process-startup)
- [Breakpoints and catchpoints](#breakpoints-and-catchpoints)
- [Code execution](#code-execution)
- [Signals](#signals)
- [State Control](#state-control)
- [Process information](#process-information)
- [Threads](#threads)
- [Shared libs](#shared-libs)
- [Stack](#stack)
- [Code and Assembler](#code-and-assembler)
- [Memory](#memory)
- [Expressions \(variables, registers, etc.\)](#expressions-variables-registers-etc)
- [Extensions](#extensions)
- [Python interpreter](#python-interpreter)
- [GUI / CUI](#gui-cui)
<!-- /MarkdownTOC -->
Configuration
-------------
### .gdbinit
It's worth enabling the following elements permanently in the `~/.gdbinit` file:
```shell
# show disassembly on every stop and use intel syntax
set disassembly-flavor intel
set disassemble-next-line on
# enable debuginfod
set debuginfod enable on
# stop on forking and exceptions
catch fork
catch vfork
catch throw
catch rethrow
```
We may check the debuginfod settings in GDB:
```
(gdb) show debuginfod
debuginfod enabled: Debuginfod functionality is currently set to "ask".
debuginfod urls: Debuginfod URLs have not been set.
debuginfod verbose: Debuginfod verbose output is set to 1.
```
### ptrace capability
To ptrace any process, you may add ptrace capability to gdb:
```shell
sudo setcap cap_sys_ptrace=eip $(which gdb)
```
TUI
---
This is a windowed interface for GDB. We enable/disable it using **Ctrl-x a**. **Ctrl-x 1** enables single-window mode, **Ctrl-x 2** enables two-window mode. The `tui layout` command determines what appears in the windows, e.g.:
```shell
tui layout split
tui layout src
```
**Ctrl-x o** allows us to switch between active debugger windows.
Symbols
-------
### Searching for symbols and addresses
`info address` finds the symbol associated with a given memory address, `info symbol` finds the address associated with a given symbol, for example:
```shell
info address lo_getattr
# Symbol "lo_getattr" is a function at address 0x555555556af0.
info symbol 0x555555556af0
# lo_getattr in section .text of /tmp/passthrough-minimal/passthrough_ll
```
`info types` searches for type declarations (accepts regexes). For functions we have `info functions` e.g.:
```shell
info functions statx
# All functions matching regular expression "statx":
#
# File ../sysdeps/unix/sysv/linux/statx.c:
# 25: int statx(int, const char *, int, unsigned int, struct statx *);
#
# File ./statx_generic.c:
# 42: static int statx_generic(int, const char *, int, struct statx *, unsigned int);
```
`ptype` allows viewing the definition of a given type. As a parameter we can provide either the type name or a variable of that type, e.g.:
```shell
ptype struct link_map
# type = struct link_map {
# Elf64_Addr l_addr;
# char *l_name;
# Elf64_Dyn *l_ld;
# struct link_map *l_next;
# struct link_map *l_prev;
# struct link_map *l_real;
# Lmid_t l_ns;
# ...
# }
```
`info scope` - shows symbols currently available in a given scope, e.g. for function `match_symbol`:
```shell
info scope match_symbol
# Scope for match_symbol:
# Symbol digits is optimized out.
# Symbol _itoa_word is a function at address 0x7ffff7fda11a, length 1.
# Symbol value is multi-location:
# Range 0x7ffff7fda21c-0x7ffff7fda220: a variable in $rcx
# Range 0x7ffff7fda240-0x7ffff7fda26f: a variable in $rcx
# Range 0x7ffff7fda26f-0x7ffff7fda275: a variable in $rdx
# , length 8.
# Symbol buflim is multi-location:
# Range 0x7ffff7fda21c-0x7ffff7fda220: a complex DWARF expression:
# 0: DW_OP_fbreg -109
# 3: DW_OP_stack_value
#
# Range 0x7ffff7fda220-0x7ffff7fda275: a variable in $rsi
# , length 8.
# Symbol base is multi-location:
# Range 0x7ffff7fda21c-0x7ffff7fda275: the constant 10
# , length 4.
# Symbol upper_case is multi-location:
# Range 0x7ffff7fda21c-0x7ffff7fda275: the constant 0
# , length 4.
# Symbol digits is optimized out.
```
### Searching for source code
The `directory` command allows us to add additional directories for the source code:
```shell
show directories
# Source directories searched: $cdir:$cwd
directory /tmp/openssl-0.9.8-easy_tls-orig/
# Source directories searched: /tmp/openssl-0.9.8-easy_tls-orig:$cdir:$cwd
```
Debugging child processes
-------------------------
`set detach-on-fork off` makes the debugger debug both the parent and its fork. If we don't enable this, we can decide what happens at fork using `set follow-fork-mode`. The `parent` option will cause the debugger to continue debugging the parent. The `child` option will switch to the child.
At the moment of fork, it's possible that `continue` won't work. We then need to allow both parent and child to execute simultaneously using `set schedule-multiple on`.
We can view currently debugged processes with the `info inferiors` command, and switch between them with `inferior {ID}`.
Execution Control
-----------------
### Process startup
GDB takes the debugged binary as an argument. Then we can add startup arguments with the `run` command. It also accepts stdin redirection from any file, e.g.:
```shell
r mytestapp < /tmp/test_file
```
### Breakpoints and catchpoints
`b func_name` sets a breakpoint on a function, `b file:line_num` sets a breakpoint on a line.
Additionally, we have special breakpoints called catchpoints for handling events (somewhat similar to `sxe` in WinDbg), e.g. `catch fork` to stop the debugger at fork:
```shell
catch fork
# Catchpoint 1 (fork)
info breakpoints
# Num Type Disp Enb Address What
# 1 catchpoint keep y fork
# 2 breakpoint keep y 0x00000000004044f0 in main at test.c:38
# breakpoint already hit 1 time
# Catchpoint 1 (forked process 5473), arch_fork (ctid=0x7ffff7ca8690) at ../sysdeps/unix/sysv/linux/arch-fork.h:50
# 50 ret = INLINE_SYSCALL_CALL (clone, flags, 0, NULL, ctid, 0);
# => 0x00007ffff7db9b57 <__GI__Fork+39>: 48 3d 00 f0 ff ff cmp rax,0xfffffffffffff000
# 0x00007ffff7db9b5d <__GI__Fork+45>: 77 39 ja 0x7ffff7db9b98 <__GI__Fork+104>
# set detach-on-fork off
c
# Continuing.
# [New inferior 2 (process 5473)]
# [Thread debugging using libthread_db enabled]
# Using host libthread_db library "/usr/lib/libthread_db.so.1".
```
The `catch` command alone will display available events (similar to `sx` in WinDbg).
`rb function_regex` allows setting breakpoints based on regular expressions:
```shell
rb ssl_shim::wrapped_.*
# Breakpoint 2 at 0x7ffff7f741cd: file src/lib.rs, line 308.
# fn ssl_shim::wrapped_SSL_CTX_check_private_key(*mut ssl_shim::ssl::ssl_ctx_st) -> i32;
# Breakpoint 3 at 0x7ffff7f74539: file src/lib.rs, line 406.
# fn ssl_shim::wrapped_SSL_CTX_ctrl(*mut ssl_shim::ssl::ssl_ctx_st, i32, i64, *mut core::ffi::c_void) -> i64;
# Breakpoint 4 at 0x7ffff7f73b4e: file src/lib.rs, line 124.
# fn ssl_shim::wrapped_SSL_CTX_free(*mut ssl_shim::ssl::ssl_ctx_st);
# Breakpoint 5 at 0x7ffff7f73f0d: file src/lib.rs, line 226.
# fn ssl_shim::wrapped_SSL_CTX_get_client_CA_list(*mut ssl_shim::ssl::ssl_ctx_st) -> *mut ssl_shim::ssl::stack_st_X509_NAME;
gitextract_r48gq6i_/ ├── .gitignore ├── 404.html ├── CNAME ├── Gemfile ├── LICENSE ├── README.md ├── _config.yml ├── _includes/ │ ├── footer.html │ └── head.html ├── _layouts/ │ ├── home.html │ └── posts.html ├── about.md ├── articles.md ├── assets/ │ ├── main.scss │ └── other/ │ ├── EtwMetadata.ps1.txt │ ├── WTComTrace.wprp │ ├── winapi-user32.ps1.txt │ └── windbg-install.ps1.txt ├── browserconfig.xml ├── guides/ │ ├── com-troubleshooting.md │ ├── configuring-linux-for-effective-troubleshooting.md │ ├── configuring-windows-for-effective-troubleshooting.md │ ├── diagnosing-dotnet-apps.md │ ├── diagnosing-native-windows-apps.md │ ├── ebpf.md │ ├── etw.md │ ├── gdb.md │ ├── linux-tracing.md │ ├── network-tracing-tools.md │ ├── using-withdll-and-detours-to-trace-winapi.md │ ├── windbg.md │ └── windows-performance-counters.md ├── guides.md ├── index.md ├── site.webmanifest └── tools.md
Condensed preview — 36 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (341K chars).
[
{
"path": ".gitignore",
"chars": 64,
"preview": "_site\n.sass-cache\n.jekyll-cache\n.jekyll-metadata\nvendor\ndraft_*\n"
},
{
"path": "404.html",
"chars": 419,
"preview": "---\npermalink: /404.html\nlayout: default\n---\n\n<style type=\"text/css\" media=\"screen\">\n .container {\n margin: 10px aut"
},
{
"path": "CNAME",
"chars": 10,
"preview": "wtrace.net"
},
{
"path": "Gemfile",
"chars": 1211,
"preview": "source \"https://rubygems.org\"\n# Hello! This is where you manage which Jekyll version is used to run.\n# When you want to "
},
{
"path": "LICENSE",
"chars": 18653,
"preview": "Attribution 4.0 International\n\n=======================================================================\n\nCreative Commons"
},
{
"path": "README.md",
"chars": 1118,
"preview": "\nDebug Recipes\n=============\n\nIt is a repository of my field notes collected while debugging various .NET application pr"
},
{
"path": "_config.yml",
"chars": 711,
"preview": "title: wtrace.net\nemail: contact@wtrace.net\ndescription: >- # this means to ignore newlines until \"baseurl:\"\n Tools and"
},
{
"path": "_includes/footer.html",
"chars": 594,
"preview": "<footer class=\"site-footer h-card\">\n <data class=\"u-url\" href=\"{{ \"/\" | relative_url }}\"></data>\n\n <div class=\"wrapper"
},
{
"path": "_includes/head.html",
"chars": 864,
"preview": "\n<head>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\n <meta name=\"viewport\" content"
},
{
"path": "_layouts/home.html",
"chars": 780,
"preview": "---\n---\n<!DOCTYPE html>\n<html lang=\"{{ page.lang | default: site.lang | default: \"en\" }}\">\n\n {%- include head.html -%}\n"
},
{
"path": "_layouts/posts.html",
"chars": 646,
"preview": "---\nlayout: default\n---\n\n<div class=\"home\">\n {%- if page.title -%}\n <h1 class=\"page-heading\">{{ page.title }}</h1>\n "
},
{
"path": "about.md",
"chars": 1110,
"preview": "---\nlayout: page\ntitle: About\n---\n\nI am **Sebastian Solnica**, a software engineer with more than 15 years of experience"
},
{
"path": "articles.md",
"chars": 58,
"preview": "---\nlayout: page\ntitle: Articles\nredirect_to: /guides\n---\n"
},
{
"path": "assets/main.scss",
"chars": 1423,
"preview": "---\n# Only the main Sass file needs front matter (the dashes are enough)\n---\n\n$brand-color: #CA4E07;\n$credits-color: #70"
},
{
"path": "assets/other/EtwMetadata.ps1.txt",
"chars": 4838,
"preview": "\r\n$ErrorActionPreference = \"Stop\"\r\n\r\n$MetadataFolder = \"$env:LOCALAPPDATA\\MyEtwMetadata\\ById\"\r\n$MetadataSearchByNameFold"
},
{
"path": "assets/other/WTComTrace.wprp",
"chars": 6676,
"preview": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n<WindowsPerformanceRecorder Version=\"1.0\" Author=\"Sebastian Solnica (https://wtr"
},
{
"path": "assets/other/winapi-user32.ps1.txt",
"chars": 2601,
"preview": "$ErrorActionPreference = \"Stop\"\r\n\r\nAdd-Type -TypeDefinition @\"\r\n using System;\r\n public enum GWL_EXSTYLE : int {\r\n"
},
{
"path": "assets/other/windbg-install.ps1.txt",
"chars": 1810,
"preview": "# script created by @Izybkr (https://github.com/microsoftfeedback/WinDbg-Feedback/issues/19#issuecomment-1513926394) wit"
},
{
"path": "browserconfig.xml",
"chars": 246,
"preview": "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<browserconfig>\n <msapplication>\n <tile>\n <square150x150logo"
},
{
"path": "guides/com-troubleshooting.md",
"chars": 27118,
"preview": "---\r\nlayout: page\r\ntitle: COM troubleshooting\r\ndate: 2023-04-07 08:00:00 +0200\r\nredirect_from:\r\n - /articles/com-trou"
},
{
"path": "guides/configuring-linux-for-effective-troubleshooting.md",
"chars": 964,
"preview": "---\r\nlayout: page\r\ntitle: Configuring Linux for effective troubleshooting\r\ndate: 2025-12-26 08:00:00 +0200\r\n---\r\n\r\n**Tab"
},
{
"path": "guides/configuring-windows-for-effective-troubleshooting.md",
"chars": 6925,
"preview": "---\r\nlayout: page\r\ntitle: Configuring Windows for effective troubleshooting\r\ndate: 2023-10-11 08:00:00 +0200\r\n---\r\n\r\n**T"
},
{
"path": "guides/diagnosing-dotnet-apps.md",
"chars": 44600,
"preview": "---\nlayout: page\ntitle: Diagnosing .NET applications\ndate: 2024-01-01 08:00:00 +0200\n---\n\n{% raw %}\n\n:point_right: I als"
},
{
"path": "guides/diagnosing-native-windows-apps.md",
"chars": 16348,
"preview": "---\r\nlayout: page\r\ntitle: Diagnosing native Windows applications\r\ndate: 2025-05-25 08:00:00 +0200\r\n---\r\n\r\n{% raw %}\r\n\r\n*"
},
{
"path": "guides/ebpf.md",
"chars": 6650,
"preview": "---\r\nlayout: page\r\ntitle: eBPF\r\ndate: 2025-12-22 08:00:00 +0200\r\n---\r\n\r\n{% raw %}\r\n\r\n**Table of contents:**\r\n\r\n<!-- Mark"
},
{
"path": "guides/etw.md",
"chars": 42157,
"preview": "---\nlayout: page\ntitle: Event Tracing for Windows (ETW)\ndate: 2025-10-02 08:00:00 +0200\nredirect_from:\n - /guides/usi"
},
{
"path": "guides/gdb.md",
"chars": 12190,
"preview": "---\r\nlayout: page\r\ntitle: GDB usage guide\r\ndate: 2025-05-27 08:00:00 +0200\r\n---\r\n\r\n{% raw %}\r\n\r\n\r\n**Table of contents:**"
},
{
"path": "guides/linux-tracing.md",
"chars": 4790,
"preview": "---\r\nlayout: page\r\ntitle: Linux Kernel Tracing (/sys/kernel/tracing)\r\ndate: 2025-12-22 08:00:00 +0200\r\n---\r\n\r\n{% raw %}\r"
},
{
"path": "guides/network-tracing-tools.md",
"chars": 11847,
"preview": "---\nlayout: page\ntitle: Network tracing tools\ndate: 2024-01-01 08:00:00 +0200\nredirect_from:\n - /guides/using-network-t"
},
{
"path": "guides/using-withdll-and-detours-to-trace-winapi.md",
"chars": 5780,
"preview": "---\nlayout: page\ntitle: Using withdll and detours to trace Win API calls \ndate: 2023-11-25 08:00:00 +0200\n---\n\n**Table o"
},
{
"path": "guides/windbg.md",
"chars": 88849,
"preview": "---\nlayout: page\ntitle: WinDbg usage guide\ndate: 2026-02-20 08:00:00 +0200\nredirect_from:\n - /guides/using-ttd/\n -"
},
{
"path": "guides/windows-performance-counters.md",
"chars": 11244,
"preview": "---\nlayout: page\ntitle: Windows Performance Counters\ndate: 2024-01-01 08:00:00 +0200\nredirect_from:\n - /guides/using-pe"
},
{
"path": "guides.md",
"chars": 2256,
"preview": "---\nlayout: page\ntitle: Guides\n---\n\nPlease first check the [Windows degugging configuration guide](configuring-windows-f"
},
{
"path": "index.md",
"chars": 797,
"preview": "---\ntitle: wtrace.net\ndescription: Tools and materials for software and system troubleshooting \nfeature_image: /assets/i"
},
{
"path": "site.webmanifest",
"chars": 426,
"preview": "{\n \"name\": \"\",\n \"short_name\": \"\",\n \"icons\": [\n {\n \"src\": \"/android-chrome-192x192.png\",\n "
},
{
"path": "tools.md",
"chars": 1314,
"preview": "---\nlayout: page\ntitle: Tools\n---\n\n### :feet: Tracing tools\n\n#### [wtrace](https://github.com/lowleveldesign/wtrace)\n\nA "
}
]
About this extraction
This page contains the full source code of the lowleveldesign/debug-recipes GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 36 files (320.4 KB), approximately 87.4k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.