Writing Windows Debugger - Part 2

Preface

This article is the sequel to previous article Writing basic Windows Debugger, and it is mandatory for the reader to read and understand the first part first! Without grasping what is mentioned in first part of this series, you may not understand what is presented here, nor you would be able to appreciate this stuff!

One thing I should mention, which I did not state in previous part, is that our debugger is only able to debug Native Code. Thus, an attempt to debug a .NET/managed application would fail. May be, in next article I would cover debugging the Managed Code also.

I am here to present more of intriguing aspects, in debugging parlance, which you might not know before. This would include showing the source code and the call stack, setting the breakpoints, stepping into code, attaching (our) debugger to a process, making it a default debugger and so on. Since this article presents somewhat advanced concepts in debugging, I removed "basic" word from the title!

The debugging experience would be as per with Visual C++, that includes debugging terms (Step-Into), shortcuts (F5) etc.

Table of contents:

• Start Debugging from main
• Obtain start-address of the process
• Place breakpoint instruction at start-address
• Handle breakpoint, revert instruction
• Halt debugging, wait for user action
• Continue debugging as per user's command
• CDebuggerCore - The debugging-interface class
• Debugging Actions
• Enumerating source files, and line numbers
• Placing User-Breakpoints
• Stepping through the code (Step-in, step-out etc)
• Conditional Breakpoints
• Debugging a Running Process
• Detach, Terminate or wait?
• Debugging a Crashing Process
• Manually attaching a Debugger

Let's Debug!

So, what you do when you want to debug your program? Well, mostly we hit F5 to begin debugging our applications, and the Visual Studio Debugger would halt at the places where we have placed breakpoints (or conditional breakpoints!). A "Debug Assertion Failed" message box, followed by Retry button also opens the source code, and halts there. The DebugBreak call or {int 3} instruction would also do the same. There are other approaches for debugging, you know!

Rarely or occasionally, we also start debugging from the beginning by hitting F11 (Step-Into), and the VS starts from main/wmain or WinMain/wWinMain (or _t prefixed variants of them). Well, that is the logical start address of your process where debugging begins. I call it logical since it is not the actual start address - the address also known as Entry Point of the module. For a console application, it is mainCRTStartup which calls main function, and VS Debugger starts at main. The entry-point is also applicable for DLLs too. Please have a look at /ENTRY switch for more information.

This turns out that we need to stop program execution at the entry-point location, and let user (programmer) continue debugging. Yes, I said "stop" the execution at entry-point location - the process is already started (being debugged), and unless we halt it somewhere the process would complete its startup. The following call-stack appeared as soon as I hit F11, for an MFC application - which clarifies what I mentioned.

What we need to do, to stop program execution at Entry-Point?

In a nutshell:

1. Obtain the start address of process
2. Modify the instruction at that address - replace it with breakpoint instruction, for example.
3. Handle the breakpoint event, revert the instruction with original instruction
4. Stop execution, show call stack, display registers and source code, if available.
5. Continue debugging (as per user request)

And this 5-step task is not easy, boy!

1. Obtain the Entry Point of the Process

Start Address, Entry Point and the logical* entry point (your main/WinMain) - welcome to the complex world. Before I present textual content about these terms, let me give you visual idea about it. But first thing you should understand: the first instruction at the given address is the point where execution begins, and debuggers play with that address only.

* [This term is coined by me, and is relevant to this article only!]

Here is how WinMain looks in Disassembly View in Visual Studio, with annotations for the gibberish you see:

You can launch the same view for you code by right clicking in code editor, and selecting Go To Disassembly. The Code Bytes are not shown by default (Green content above), you enable it by Show Code Bytes context menu.

Relax! You need not to understand the machine language instructions, nor the ASM language! This is just for illustration. For the example above, 00978F10 is the start-address, 8B FF is the first instruction. We just need to make the given instruction (mov blah blah), a breakpoint instruction. We know the API called DebugBreak - but that cannot be used here. {int 3} is the assembly code for the same, and we need the x86 instruction for the same. The breakpoint x86 instruction is 0xCC(204).

Thus, we just replace 8B with CC and we are done! When the program continues further (within our debugger-loop), it would raise an exception event (EXCEPTION_DEBUG_EVENT), having the exception code EXCEPTION_BREAKPOINT (0x80000003). We know this is our sin, we handle is appropriately. If you don't understand this paragraph, I request you (last time) to read the first part of this article.

The x86 instructions are not of fixed width - but who cares? We don't need to see if instruction is of 1, 2, or N bytes. We always replace the first byte. The first byte (of instruction) may be anything, and not just 8B! But, we must ensure that as soon as desired breakpoint is hit, we reverse our sin - i.e. replace the replaced instruction with original code-byte.

For few geeks out there, who know it, and for everyone else who do not know it. The breakpointing is not the only way to stop program at start-address. There is better alternative for these One-shot/stop-once breakpoints, which is to be discussed. Secondly, CC instruction is not the only breakpoint instruction - but it is sufficient