Effective minidumps (Part 2)
MiniDumpCallback function
If we want to customize the contents of our minidumps beyond the reach of MINIDUMP_TYPE flags, we can use MiniDumpCallback function. This is a user-defined callback, which is called by MiniDumpWriteDump when it needs the user’s decision on whether to include some data into the minidump. With the help of this function, we can accomplish the following tasks:
- Exclude an executable module from the minidump’s module information (completely or partially)
- Exclude a thread from the minidump’s thread information (completely or partially)
- Include the contents of a user-specified range of memory into the minidump
Lets look at the declaration of MiniDumpCallback function (shown in Figure 7):
Figure 7:
BOOL CALLBACK MiniDumpCallback( PVOID CallbackParam, const PMINIDUMP_CALLBACK_INPUT CallbackInput, PMINIDUMP_CALLBACK_OUTPUT CallbackOutput );
The function receives three parameters. The first parameter (CallbackParam) is a user-defined context for the callback function (for example, a pointer to a C++ object). The second parameter (CallbackInput) contains the data that MiniDumpWriteDump passes to the callback. The third parameter (CallbackOutput) contains the data that is returned by the callback to MiniDumpWriteDump (and this data usually specifies what information should be included into the minidump).
Now lets look at the contents of MINIDUMP_CALLBACK_INPUT and MINIDUMP_CALLBACK_OUTPUT structures.
Figure 8:
typedef struct _MINIDUMP_CALLBACK_INPUT { ULONG ProcessId; HANDLE ProcessHandle; ULONG CallbackType; union { HRESULT Status; MINIDUMP_THREAD_CALLBACK Thread; MINIDUMP_THREAD_EX_CALLBACK ThreadEx; MINIDUMP_MODULE_CALLBACK Module; MINIDUMP_INCLUDE_THREAD_CALLBACK IncludeThread; MINIDUMP_INCLUDE_MODULE_CALLBACK IncludeModule; }; } MINIDUMP_CALLBACK_INPUT, *PMINIDUMP_CALLBACK_INPUT; typedef struct _MINIDUMP_CALLBACK_OUTPUT { union { ULONG ModuleWriteFlags; ULONG ThreadWriteFlags; struct { ULONG64 MemoryBase; ULONG MemorySize; }; struct { BOOL CheckCancel; BOOL Cancel; }; HANDLE Handle; }; } MINIDUMP_CALLBACK_OUTPUT, *PMINIDUMP_CALLBACK_OUTPUT; typedef enum _MINIDUMP_CALLBACK_TYPE { ModuleCallback, ThreadCallback, ThreadExCallback, IncludeThreadCallback, IncludeModuleCallback, MemoryCallback, CancelCallback, WriteKernelMinidumpCallback, KernelMinidumpStatusCallback, } MINIDUMP_CALLBACK_TYPE;
MINIDUMP_CALLBACK_INPUT structure formulates the MiniDumpWriteDump’s request to the callback function. The meaning of the first two members is obvious – they contain the id and handle of the process the minidump is created for. The third member (CallbackType) contains the type of the request, naturally called the callback type. All possible values of CallbackType are collected in MINIDUMP_CALLBACK_TYPE enumeration (which is also shown in Figure 8) and we will take a closer look at them soon. The fourth member of the structure is a union, whose meaning varies depending on the value of CallbackType. The union contains additional data about the MiniDumpWriteDump’s request.
MINIDUMP_CALLBACK_OUTPUT structure is a bit simpler. It consists of a union, whose interpretation also depends on the value of MINIDUMP_CALLBACK_INPUT.CallbackType. Members of the union contain the callback’s response to the MiniDumpWriteDump’s request.
Now it’s time to walk through the list of the most important requests (identified by callback type) and see how the callback function can respond to them. But before we do it, take a look at Figure 9 – it shows how to tell MiniDumpWriteDump that a user-defined callback function exists and should be called.
Figure 9:
void CreateMiniDump( EXCEPTION_POINTERS* pep ) { // Open the file HANDLE hFile = CreateFile( _T("MiniDump.dmp"), GENERIC_READ | GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL ); if( ( hFile != NULL ) && ( hFile != INVALID_HANDLE_VALUE ) ) { // Create the minidump MINIDUMP_EXCEPTION_INFORMATION mdei; mdei.ThreadId = GetCurrentThreadId(); mdei.ExceptionPointers = pep; mdei.ClientPointers = FALSE; MINIDUMP_CALLBACK_INFORMATION mci; mci.CallbackRoutine = (MINIDUMP_CALLBACK_ROUTINE)MyMiniDumpCallback; mci.CallbackParam = 0; // this example does not use the context MINIDUMP_TYPE mdt = MiniDumpNormal; BOOL rv = MiniDumpWriteDump( GetCurrentProcess(), GetCurrentProcessId(), hFile, mdt, (pep != 0) ? &mdei : 0, 0, &mci ); if( !rv ) _tprintf( _T("MiniDumpWriteDump failed. Error: %u \n"), GetLastError() ); else _tprintf( _T("Minidump created.\n") ); // Close the file CloseHandle( hFile ); } else { _tprintf( _T("CreateFile failed. Error: %u \n"), GetLastError() ); } } BOOL CALLBACK MyMiniDumpCallback( PVOID pParam, const PMINIDUMP_CALLBACK_INPUT pInput, PMINIDUMP_CALLBACK_OUTPUT pOutput ) { // Callback implementation … }
IncludeModuleCallback
With the callback type set to IncludeModuleCallback, MiniDumpWriteDump asks the callback function whether the information about a particular executable module should be included into the minidump. The callback bases its decision on the contents of MINIDUMP_CALLBACK_INPUT structure, whose union member is interpreted as MINIDUMP_INCLUDE_MODULE_CALLBACK:
typedef struct _MINIDUMP_INCLUDE_MODULE_CALLBACK { ULONG64 BaseOfImage; } MINIDUMP_INCLUDE_MODULE_CALLBACK, *PMINIDUMP_INCLUDE_MODULE_CALLBACK;
Here, BaseOfImage is the base address of the module in memory, which can be used to obtain more information about the module and decide whether it is needed in the minidump.
The callback function uses its return value to communicate the decision to MiniDumpWriteDump. If the callback returns TRUE, the information about the module will be included into the minidump (and the exact contents of this information can be further customized in subsequent calls to the callback function). If the callback returns FALSE, the information about the module will be discarded, and no trace of the module’s existence will be seen in the minidump.
MINIDUMP_CALLBACK_OUTPUT structure is not used with this callback type.
ModuleCallback
After a module has passed the test in IncludeModuleCallback and survived, it faces another hurdle on the way into the minidump. This hurdle is ModuleCallback, where the callback function can decide what kinds of information about the module should be included.
This time, the return value of the callback function must be TRUE (in order to let MiniDumpWriteDump continue), and MINIDUMP_CALLBACK_OUTPUT structure is used to communicate the callback’s decisions to MiniDumpWriteDump. The union inside the structure is interpreted in favour of ModuleWriteFlags member, which contains a set of flags initialized by MiniDumpWriteDump. These flags represent various kinds of module information that can be included into the minidump, and currently existing flags are collected in MODULE_WRITE_FLAGS enumeration (see Figure 10).
Figure 10:
typedef enum _MODULE_WRITE_FLAGS { ModuleWriteModule = 0x0001, ModuleWriteDataSeg = 0x0002, ModuleWriteMiscRecord = 0x0004, ModuleWriteCvRecord = 0x0008, ModuleReferencedByMemory = 0x0010, ModuleWriteTlsData = 0x0020, ModuleWriteCodeSegs = 0x0040, } MODULE_WRITE_FLAGS;
When MiniDumpWriteDump is calling the callback function with ModuleCallback callback type, it sets some of the flags, telling the callback what kinds of module information can be included into the minidump. The callback function can analyze the flags and decide to clear some of them (or even all of them), in turn telling MiniDumpWriteDump what kinds of information should not be included. The table in Figure 11 lists the currently available flags, and describes what kinds of information they represent.
Figure 11:
Flag | Description |
---|---|
ModuleWriteModule |
This flag allows to exclude from the minidump all kinds of information about the module. If the callback
clears this flag, all other flags will be ignored, and no information about the module will be available
in the minidump. By default, this flag is always set. |
ModuleWriteCvRecord, ModuleWriteMiscRecord |
These flags allow to exclude the module’s debug information record from the minidump. If these flags are cleared,
the debugger will be able to load debug information for the module only if the module itself is also available
on the developer’s machine. By default, these flags are set if the module contains debug information record of the given kind (which in turn is present only if the module was built with debug information). You can find more information about debug information records in this article. |
ModuleWriteDataSeg |
This flag allows to exclude the contents of the module’s data sections from the minidump. It can be very useful
when we want to use MiniDumpWithDataSegs flag with MiniDumpWriteDump, but also want to choose the modules
whose data sections should be included. (Usually we want to see the data sections of all modules written
by ourselves (so that we can inspect their global variables in the debugger), and also a small set
of system modules (e.g. ntdll.dll), but do not need the data sections of other system and third party modules).
Since data sections of executable modules can occupy significant space in the minidump, this flag gives us
a good opportunity of size optimisation. This flag is only set if MiniDumpWithDataSegs flag has been passed to MiniDumpWriteDump. |
ModuleWriteCodeSegs | This flag allows to exclude the contents of the module’s code sections from the minidump. It is set (and therefore can be used) only if MiniDumpWithCodeSegs flag has been passed to MiniDumpWriteDump function. This flag is useful if we want to choose the modules whose code sections should be included into the minidump (but do not want to include code sections of all modules, which could significantly increase the size of the minidump). |
ModuleReferencedByMemory |
This flag is used together with MiniDumpScanMemory flag from MINIDUMP_TYPE enumeration.
If the latter flag has been passed to MiniDumpWriteDump, the function will scan the stacks
of all threads in the process, looking for pointers that point into the address ranges occupied
by executables modules. After the scan has completed, MiniDumpWriteDump knows which modules
are referenced from the thread stacks and which are not. If no one of the thread stacks references a module, it is very likely that the module is not needed to recover the call stacks, and information about this module can be excluded from the minidump (thus saving space). In order to let the callback function make the final decision, MiniDumpWriteDump sets ModuleReferencedByMemory flag for every module that is referenced from the stack, and clears this flag for all modules that are not referenced. In turn, the callback function can check the flag to see whether the module is referenced, and exclude the module from the minidump (by clearing its ModuleWriteModule(!) flag) if it is not. |
ModuleWriteTlsData | This flag is probably supposed to control whether the module’s TLS data (allocated via __declspec(thread)) should be included into the minidump. But at the time of this writing I could not make it work. |
Note that ModuleCallback only allows us to exclude some parts of module information, but does not allow to add new data. It means that if a flag is not set by MiniDumpWriteDump, setting it in the callback function has no effect. For example, if we do not pass MiniDumpWithDataSegs flag to MiniDumpWriteDump, it will not set ModuleWriteDataSeg flag for any of the modules. Then, even if the callback function will set ModuleWriteDataSeg flag for a module, the contents of the module’s data sections will not be included into the minidump.
After such a long excursion into the contents of MINIDUMP_CALLBACK_OUTPUT structure, lets pay attention to MINIDUMP_CALLBACK_INPUT. It is interesting that this time the union is interpreted as MINIDUMP_MODULE_CALLBACK structure (Figure 12), which contains a rich set of information about the module (e.g. the name and path, size, version information).
Figure 12:
typedef struct _MINIDUMP_MODULE_CALLBACK { PWCHAR FullPath; ULONG64 BaseOfImage; ULONG SizeOfImage; ULONG CheckSum; ULONG TimeDateStamp; VS_FIXEDFILEINFO VersionInfo; PVOID CvRecord; ULONG SizeOfCvRecord; PVOID MiscRecord; ULONG SizeOfMiscRecord; } MINIDUMP_MODULE_CALLBACK, *PMINIDUMP_MODULE_CALLBACK;
IncludeThreadCallback
This callback type plays the same role for threads as IncludeModuleCallback does for modules – it gives us the opportunity to decide whether the information about a thread should be included into the minidump. As with IncludeModuleCallback, the callback function should return TRUE to include the information about the thread into the minidump, or FALSE to discard this information completely. The thread can be identified by its system identifier, which is stored in MINIDUMP_CALLBACK_INPUT’s union as the following:
typedef struct _MINIDUMP_INCLUDE_THREAD_CALLBACK { ULONG ThreadId; } MINIDUMP_INCLUDE_THREAD_CALLBACK, *PMINIDUMP_INCLUDE_THREAD_CALLBACK;
MINIDUMP_CALLBACK_OUTPUT structure is not used.
ThreadCallback
This callback type serves the same purpose for threads as ModuleCallback does for modules, and the basic principles behind both callback types are also the same. The union in MINIDUMP_CALLBACK_OUTPUT structure is interpreted as a set of flags (ThreadWriteFlags), and the callback function can clear some (or all) of them to exclude the corresponding parts of thread information from the minidump.
A rich set of information about the thread is provided in MINIDUMP_CALLBACK_INPUT structure, whose union is interpreted as MINIDUMP_THREAD_CALLBACK (Figure 13). The information includes the thread’s identifier and handle, thread context, and boundaries of the thread’s stack. The callback function must return TRUE to let MiniDumpWriteDump continue.
Figure 13:
typedef struct _MINIDUMP_THREAD_CALLBACK { ULONG ThreadId; HANDLE ThreadHandle; CONTEXT Context; ULONG SizeOfContext; ULONG64 StackBase; ULONG64 StackEnd; } MINIDUMP_THREAD_CALLBACK, *PMINIDUMP_THREAD_CALLBACK;
The table in Figure 14 lists the most important flags and describes what kinds of information they represent.
Figure 14:
Flag | Description |
---|---|
ThreadWriteThread |
This flag allows to exclude from the minidump all kinds of information about the thread. If the callback function
clears this flag, all other flags are ignored, and no information about the thread will be available in the minidump. By default, this flag is always set. |
ThreadWriteStack |
This flag allows to exclude from the minidump the contents of the thread’s stack. Therefore, if the callback
clears this flag, it will not be possible to see the call stack of the thread in the debugger. Thread stacks
usually occupy from several kilobytes (more often) to several megabytes (rarely) of memory, thus the potential
impact of this flag on the size of the minidump. By default, this flag is always set. |
ThreadWriteContext |
This flag allows to exclude from the minidump the contents of the thread’s context (platform-specific CONTEXT
structure, defined in winnt.h). If the callback function clears this flag, the debugger will not be able
to show the thread’s context (all registers will be set to zero) and the call stack. Thread contexts do not occupy too much space in the minidump (716 bytes on x86), therefore the impact of this flag on the size of the minidump is small. By default, this flag is always set. |
ThreadWriteInstructionWindow | This flag allows to exclude from the minidump the contents of the thread’s instruction window (256 bytes of memory around the current instruction pointer). If this flag is cleared, it will be possible to see the disassembly of the code the thread was executing at the moment of failure only if the code belongs to a module, and the module itself is available on the developer’s machine. |
ThreadWriteThreadInfo | This flag is set only if MiniDumpWithThreadInfo flag was passed to MiniDumpWriteDump function, and it allows to exclude the additional information about the thread from the minidump (see MiniDumpWithThreadInfo description in this article for more information). |
ThreadWriteThreadData | This flag is set only if MiniDumpWithProcessThreadData flag was passed to MiniDumpWriteDump, and it allows to exclude the thread specific information from the minidump (the contents of the thread’s environment blocks (TEBs), TLS slots, and some additional data). |
MemoryCallback
Sometimes we would like to include the contents of some additional memory regions into the minidump. For example, if we allocate some data on the heap (or simply by VirtualAlloc) and want to see that data when debugging the minidump. We can accomplish it with the help of MemoryCallback callback type, which is called by MiniDumpWriteDump after it has collected the thread and module information using the callback types described above.
When the callback function is called with MemoryCallback as the callback type, the union inside MINIDUMP_CALLBACK_OUTPUT structure is interpreted as the following:
struct { ULONG64 MemoryBase; ULONG MemorySize; };
If the callback function fills the members of this structure with the address and size of a readable memory block and returns TRUE, the contents of the memory block will be saved in the minidump. It is possible to specify more than one block, because if the callback function returned TRUE, it will be called again (with MemoryCallback callback type, of course). MiniDumpWriteDump will stop calling the callback function only after it returned FALSE.
CancelCallback
This callback type (which is called by MiniDumpWriteDump periodically) allows to cancel the process of creating a minidump, which can be useful in GUI applications. The union in MINIDUMP_CALLBACK_OUTPUT structure is interpreted as two values, Cancel and CheckCancel:
struct { BOOL CheckCancel; BOOL Cancel; };
If we want to cancel the minidump creation completely, we should set Cancel to TRUE. If we do not want to cancel it now but want to receive CancelCallback callbacks later, CheckCancel should be set to TRUE. If both members are set to FALSE, MiniDumpWriteDump will not call the callback function with CancelCallback callback type anymore.
The callback function should return TRUE to confirm the values it set in MINIDUMP_CALLBACK_OUTPUT structure.
The order of callbacks
After we have discussed the callback types, it can be interesting to see the order in which they are used by MiniDumpWriteDump. Here is the order:
- IncludeThreadCallback – once for every thread in the process
- IncludeModuleCallback – once for every executable module in the process
- ModuleCallback – once for every module that was not excluded by IncludeModuleCallback
- ThreadCallback – once for every thread that was not excluded by IncludeThreadCallback
- MemoryCallback is called one or more times, until the callback function returns FALSE
In addition, CancelCallback type is used periodically in between of other callback types, so that it is possible to cancel the minidump creation process if necessary.
This sample application allows to see the order in action. You can also use MiniDump Wizard application (described below) to experiment with callbacks.
MiniDump Wizard
You can use MiniDump Wizard application to experiment with various minidump options and see how they affect the size and contents of minidumps. MiniDump Wizard can create a minidump for an arbitrary process, and it is also possible to simulate an exception and create a minidump for MiniDump Wizard itself. You can choose what minidump type flags will be passed to MiniDumpWriteDump function, and respond to callback requests in a series of dialogs.
After the minidump has been created, load it into debugger and check what kinds of information are available. You can also use MiniDumpView application to get the list of all kinds of information available in the minidump.
User data streams
While the application state captured by MiniDumpWriteDump is essential for successful debugging, we often need additional information about the environment the application was running in. For example, it can be useful to look at the contents of a configuration file, or check the application-specific settings in Registry. MiniDumpWriteDump allows us to include this information into the minidump as additional data streams.
This sample application shows how to do it. We should declare a variable of type MINIDUMP_USER_STREAM_INFORMATION and fill its contents with the number of streams and a pointer to the array of user data streams. Every user data stream is described by MINIDUMP_USER_STREAM structure, which contains the stream type (it serves as a unique identifier of the stream, and must be larger than LastReservedStream constant), size and a pointer to the stream data. The structures are shown in Figure 14.
Figure 14:
typedef struct _MINIDUMP_USER_STREAM_INFORMATION { ULONG UserStreamCount; PMINIDUMP_USER_STREAM UserStreamArray; } MINIDUMP_USER_STREAM_INFORMATION, *PMINIDUMP_USER_STREAM_INFORMATION; typedef struct _MINIDUMP_USER_STREAM { ULONG32 Type; ULONG BufferSize; PVOID Buffer; } MINIDUMP_USER_STREAM, *PMINIDUMP_USER_STREAM;
After we have added a user data stream to the minidump, we can read it with the help of MiniDumpReadDumpStream function. This sample application shows how to do it, using the sample data written to the dump by the previous sample.
Strategies
The rich nature of MiniDumpWriteDump and the large set of available options make it difficult to choose the strategy that will work equally well for all kinds of applications. In every particular case, developers of the application have to decide what options will be useful for their debugging tasks. Here I will try to describe some basic strategies that show how the knowledge of MiniDumpWriteDump configuration options can be applied in real world scenarios. We will look at four different approaches to collecting data with MiniDumpWriteDump, and see how they affect the size of the minidump and the possibility of effective debugging.
TinyDump
This is not actually a real world scenario. Instead, this approach shows the smallest possible set of data that can be included into the minidump to keep it at least slightly useful. MiniDumpWriteDump configuration options are summarized in the table in Figure 15.
Figure 15:
MINIDUMP_TYPE flags | MiniDumpNormal |
MiniDumpCallback |
IncludeThreadCallback – exclude all threads IncludeModuleCallback – exclude all modules |
The sample application realizing this approach can be found here. The resulting minidump is extremely small, only 2 kilobytes on my system. Not a surprise, because we excluded all thread and module information. If you try to load the minidump into WinDbg or VS.NET debugger, you will see that the debugger cannot load it.
But the minidump is not completely useless, because it still contains the information about the exception. We can read this information manually (using MiniDumpReadDumpStream function) and see the address where the exception occurred, thread context at the moment of the exception, exception code, and even disassembly. You can use MiniDumpView tool to display this information (except for disassembly, to keep it simple).
MiniDump
Unlike TinyDump, this approach can be used in real world scenarios to collect enough information for debugging and still keep the minidump size as small as possible. The table in Figure 16 describes MiniDumpWriteDump configuration options.
Figure 16:
MINIDUMP_TYPE flags |
MiniDumpWithIndirectlyReferencedMemory, MiniDumpScanMemory |
MiniDumpCallback |
IncludeThreadCallback – include all threads IncludeModuleCallback – include all modules ThreadCallback – include all threads ModuleCallback – check ModuleWriteFlags and exclude all modules whose ModuleReferencedByMemory flag is not set (clear ModuleWriteModule flag for such modules) |
The sample application can be found here. The resulting minidump is still small (between 40 and 50 kilobytes on my system), but it is definitely more informative than a minidump created with the standard approach (MiniDumpNormal + no MiniDumpCallback), because it allows to look at the data referenced from the stack. As a size optimisation, we exclude from the minidump all modules which are not referenced from the thread stacks (on my system, the following modules are excluded – advapi32.dll and rpcrt4.dll).
But the minidump still lacks various kinds of important information. For example, we cannot see the values of global variables, and cannot inspect the data allocated on the heap and in TLS (unless it is referenced from the thread stacks).
MidiDump
The next approach will result in a very informative minidump, whose size will still be kept far from becoming huge. The table in Figure 17 describes this configuration.
Figure 17:
MINIDUMP_TYPE flags |
MiniDumpWithPrivateReadWriteMemory, MiniDumpWithDataSegs, MiniDumpWithHandleData, MiniDumpWithFullMemoryInfo, MiniDumpWithThreadInfo, MiniDumpWithUnloadedModules |
MiniDumpCallback |
IncludeThreadCallback – include all threads IncludeModuleCallback – include all modules ThreadCallback – include all threads ModuleCallback – exclude data sections of all modules except the main executable and ntdll.dll |
The sample application can be found here. The size of the minidump is about 1350 kilobytes on my system. When loading it into a debugger, we can obtain almost all kinds of information about the application, including the values of global variables, the contents of heaps and TLS, PEB, TEBs. We can even obtain handle information and inspect the virtual memory layout. A very useful dump, indeed, and it is not too big because we still care about its size – the following information is not included:
- Code sections of all modules (because we do not need them if we can obtain the modules themselves)
- Data sections of some modules (we include data sections only for the modules whose global variables we really want to see in the debugger)
MaxiDump
The final example shows how to create a minidump that includes all possible sets of information. The table in Figure 18 shows how it can be done.
Figure 18:
MINIDUMP_TYPE flags |
MiniDumpWithFullMemory, MiniDumpWithFullMemoryInfo, MiniDumpWithHandleData, MiniDumpWithThreadInfo, MiniDumpWithUnloadedModules |
MiniDumpCallback | Not used |
The sample application can be found here. The minidump is large (8 megabytes on my system) even for such a simple application. But it gives us access to all possible kinds of information that can be included into a minidump.
Comparison
The table in Figure 19 compares the sizes of minidumps created with all four approaches. In addition to the data collected with sample applications (which are far from being realistic, of course), I also included the data collected for a real production application that actively uses all kinds of data that can be found in a minidump.
Figure 19:
TinyDump | MiniDump | MidiDump | MaxiDump | |
---|---|---|---|---|
Sample application | 2 KB | 40-50 KB | 1,35 MB | 8 MB |
Real application | 2 KB | 200 KB | 14 MB | 35 MB |
Miscellaneous
A note about 64 bits
Some configuration options of MiniDumpWriteDump are not discussed in this article because they are intended for use on 64-bit systems. Since I don’t currently have 64-bit machines in my lab, I cannot yet provide any useful information about them.
A note about DbgHelp version
DbgHelp.dll is constantly evolving, and new features appear together with new releases of Debugging Tools for Windows package. At the time of this writing, I used DbgHelp.dll 6.3, which was the latest officially released version.
Sample applications
Sample applications for the article (with build instructions) can be found here.
Contact
Have questions or comments? Free free to contact Oleg Starodumov at firstname@debuginfo.com.