|
Updated: 21.06.2005
Dr. Watson and its limitations
NTSD - a better JIT debugger
JIT debugging under any user account
Crash dump files with unique names
Customizing the minidump format
Sending notifications
Troubleshooting NTSD startup
Dr. Watson strikes back
Just-in-time debugger registration
Dr. Watson is probably the most popular just-in-time (JIT) debugger nowadays. When an application
crashes, it is usually Dr. Watson who is called on to create the crash log and dump files. No doubt,
the information collected by Dr. Watson is useful, and usually it allows us to find the reason of
the problem. But can we call Dr. Watson an ideal JIT debugger? I don't think so. Here is the list
of limitations, which are (in my opinion) are serious enough to start looking for a better JIT debugger.
1) Dr. Watson cannot debug applications that are running under non-administrative accounts
(e.g. it cannot debug applications running under LocalService and NetworkService accounts).
In brief, it happens because when kernel32!UnhandledExceptionFilter function calls CreateProcess
to start just-in-time debugger (Dr. Watson in this case), it forces the debugger process to attach
to WinSta0 window station and WinSta0\Default desktop. By default, only LocalSystem account and
members of Administrators group have access to these objects, all other user accounts cannot
access them. As a result, any application that uses windows (I mean User objects) or console
cannot operate properly and fails in early startup phase.
2) Dr. Watson cannot send notifications over network (and thus does not provide a way for us
to get notified when an application crashes on a remote system).
3) Dr. Watson cannot create crash dump files with unique names (it always uses the file name
specified in its settings, and every subsequent debugging session will overwrite the old crash
dump file with the new one). As a result, we can lose an important crash dump if another application
happens to crash before we have managed to copy the dump file into a safe location.
4) There is a limited choice of crash dump formats. On Windows NT 4.0 and Windows 2000, only one
crash dump format is available (so called “full user dump”). Windows XP and Windows Server 2003
support two additional formats (standard minidump and minidump with full memory contents).
If you read my Effective Minidumps article, you already know that
better choices of minidump formats often exist.
Well, where can we find a better JIT debugger? In Debugging Tools for Windows,
of course! The Debugging Tools package includes NTSD debugger, which can easily overcome all the limitations
we have just discussed.
Let's take a look at NTSD command line options and see how we can configure it for JIT debugging.
(If you don't remember how JIT debuggers are registered, here is a brief introduction).
In its simplest possible form, NTSD's JIT debugging capabilities are similar to Dr. Watson's:
c:\dbgtools\ntsd.exe -p %ld -e %ld -g -c ".dump c:\dumps\jit.dmp;q"
This command line asks NTSD to attach to the failed process, create a standard minidump and save it
in c:\dumps\jit.dmp file. After the dump has been created, q command asks NTSD to exit.
Now let's add more command line options, to overcome the limitations of Dr. Watson.
-noio command line option allows NTSD to JIT-debug processes running under any user account.
When this option is specified, NTSD does not create a console (and does not access windows either),
which allows it to start successfully under user accounts that do not have access to WinSta0.
Here is the enhanced command line:
c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump c:\dumps\jit.dmp;q"
Note that this option is only supported since Debugging Tools 6.4.
If we use /u option with .dump command, the command will create a crash dump with unique name
(based on the specified file name, current date and time, and some additional information;
for example, jit_0648_2005-06-13_23-42-49-834_0638.dmp). Here is the new command line:
c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump /u c:\dumps\jit.dmp;q"
Other options of .dump command allow to customize the format and contents of the minidump.
Complete list can be found in Debugging Tools' documentation (search for .dump command),
and here I will show the most popular ones:
- /f
- Full user dump (old format, not recommended)
- /m
- Standard minidump (equivalent of MiniDumpNormal minidump type; this option is used by default)
- /ma
- Minidump with all possible options (memory, handles, unloaded modules, etc.)
- /mFhutwd
- Minidump with data sections, non-shared read/write memory pages and other useful information (my personal favorite when size of the minidump is important)
You can find more information about contents of minidumps in this article.
And here is the new command line that creates a minidump with as much information as possible:
c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump /ma /u c:\dumps\jit.dmp;q"
Finally, let's notify ourselves when an application crashes on another system in the network
(e.g. in test lab). NTSD makes it possible with the help of !net_send command, which allows
to send a message to another computer over the network. This command line takes the following parameters:
!net_send SenderMachine TargetMachine SenderMachine MessageText
(more details can be found in Debugging Tools' documentation)
Assuming that NTSD is running on the computer called TestPc, and we want to send a message to
the computer called DevPc, the command would look like this:
!net_send TestPc DevPc TestPc MessageText
And here is the final command line for NTSD JIT debugger:
c:\dbgtools\ntsd.exe -p %ld -e %ld -g -noio -c ".dump /ma /u c:\dumps\jit.dmp;!net_send TestPc DevPc TestPc Crash dump created;q"
When I register NTSD as the JIT debugger, I sometimes make typing mistakes that do not allow NTSD to work
or even start properly. Since I configure NTSD to work without displaying its console (-noio option)
and to exit immediately after debugging has been completed (q command at the end), it is difficult
to determine the reason of the failure. There is a simple solution – remove -noio option and q command
and test NTSD with an application running under an interactive user account. For example:
c:\dbgtools\ntsd.exe -p %ld -e %ld -g -c ".dump /ma /u c:\dumps\jit.dmp"
Now, when NTSD starts, it displays the console window, and we are able to see the error messages
and find the reason of the problem.
Note that there is one situation where the method described here cannot help – when
the failing application is running under an account that does not have write access
to the directory where we store crash dump files (c:\dumps in the examples).
Make sure that the directory is accessible for all potential “clients” of the JIT
debugger.
It looks like NTSD is a very attractive option for JIT debugging. It really is, but at the same time
Dr. Watson has one serious advantage – it is present on every end user system, while Debugging Tools
are not. (Curiously, NTSD is also present on all NT-based operating systems (in System32 directory),
but on most versions of Windows it's an older version, which does not support most of the options
and commands described here). Nevertheless, NTSD as part of Debugging Tools is a serious contender
when we have control over configurations of end user systems, for example in test labs or on in-house servers.
Here is a brief tour of JIT debugger registration and configuration settings.
When an application crashes (raises an unhandled exception), kernel32!UnhandledExceptionFilter function
looks for the command line of the currently registered JIT debugger in the following Registry entry:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug
Debugger = REG_SZ
The command line must have the following format:
debugger.exe -p %ld -e %ld [DebuggerSpecificOptions]
%ld options are printf-like wildcards. Kernel32!UnhandledExceptionFilter replaces the first wildcard
with the process ID of the process that will be debugged, and the second wildcard is replaced with
the handle of a special event object that the debugger will signal after it has completed its work.
Other options (DebuggerSpecificOptions) are optional, and differ for various existing debuggers.
Here is the command line used by Dr. Watson by default:
drwtsn32 -p %ld -e %ld -g
There is also another important Registry entry related to JIT debugging:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug
Auto = REG_SZ
If this entry is set to “1”, kernel32!UnhandledExceptionFilter will launch the JIT debugger immediately.
If this entry is set to “0”, the function will display a dialog asking the user if he wants to debug or
just terminate the failed application. In unattended JIT debugging scenarios (as the ones discussed in
this article) this entry should be set to “1”.
|
