Fine-tuned Windows scenarios: debugger-assisted recording with WinDbg


Jun 16, 2022
by Mathieu
Categories: Tutorial -
Tags: Reverse Engineering - REVEN - Scenario recording - WinDbg - VMI - Malware -




In this article we present a new way to leverage WinDbg to interactively control the VM during the recording of a scenario: this enables the user to more intuitively get to the point of interest and check conditions to control the execution, resulting in a more precise recording.

You can also check the documentation about the WinDbg integration in REVEN in general.

The situation

Making recordings shorter is a key skill when using REVEN. Among the tools and techniques available so far were the automatic binary recording and the ASM stub, which enable introspection and automation during this preliminary phase. In a previous article, we presented an interactive method to control the state of the VM, and thus the recording, from the hypervisor side using QEMU’s native GDB-server mode.

The latter is very versatile and is our go-to method in a number of different situations. However, since neither QEMU nor GDB are aware of the nature of the running kernel, it is also cumbersome: the absence of symbols requires creative workarounds, such as recording and replaying extremely short traces along the way to fall back on REVEN’s OS Specific Information (OSSI).

On the Windows side this led us to look at WinDbg, which can connect to a distant machine or VM and propose a native kernel debugging environment. Unfortunately, it does not seem to be possible to use the modern, fast Ethernet debugger connection with QEMU’s emulated hardware. Falling back on legacy serial debugger connection is possible, but our testing shows it is uncomfortably slow.

As a result we extended the Virtual Machine Instrospection (VMI) tools that powered the automatic binary recording to allow the REVEN-Kd Bridge to connect to a live running VM (whereas the existing WinDbg integration connects to a trace).

Example 1: recording a browser crash

Let us consider the same example as the GDB article: we want to start a recording on a call to chrome!net::HttpBasicStream::SendRequest and stop it when ntdll!KiUserExceptionDispatch is called (when the browser tab crashes). As we will see, using WinDbg for this task is more straightforward because the symbols are resolved automatically.

Connecting

First, let’s restore a VM snapshot where everything is ready for recording. Then, we can start the WinDbg VMI backend:

VMI backend started

Next, we connect the bridge to this backend, as we usually do when working with WinDbg on REVEN traces:

Connect bridge on backend

NOTE: even though we seem to connect WinDbg to a serial connection, this is actually passing through a named pipe and a network connection to the server — the displayed baud rate does not apply.

Finally, we connect WinDbg to this bridge. Everything is now setup and ready:

Connect WinDbg

Usage

We can now control the VM through WinDbg. For our use case we must:

  • Find the chrome process.
  • Place the entry break point.
  • Resume the VM.

We will use the following commands (see the 2.11.0 documentation for more details):

dx @$cursession.Processes.Where(p => p.Name.ToLower().Contains("chrome")).First().SwitchTo()
.reload /user
bp chrome!net::HttpBasicStream::SendRequest
g

We now head over the VM and click on the exploit’s link. The VM breaks:

WinDbg chrome breakpoint

At that point, since the VM is paused, we can prepare the next steps in order:

  • Disable the existing breakpoint as we don’t need it anymore
  • Place a breakpoint on the crash handler.

We use the following commands:

bd 0
bp ntdll!KiUserExceptionDispatch

We now start the recording on the Project Manager’s page: the recording will actually start when the VM is resumed.

WinDbg start recording

And we can now resume the VM with g. After a while the browser crashed, and the VM breaks again.

WinDbg break at crash

The breakpoint is set on the whole VM, so for good measure we should check that the crashed process is our target:

WinDbg break at crash

We are in the expected process. If not, we could have resumed the VM and the recording would have resumed as well. In our case, the target has crashed as expected, so we can stop the recording:

WinDbg start recording

At this point, we can remove all breakpoints again and let the VM continue, again to make sure that our tab has crashed. This will not be part of the recorded trace.

Final crash

We can now confidently commit the recording. The resulting trace will precisely start & stop where we wanted to, with little effort on our part.

Here is a video of the whole process, starting with a connected WinDbg:


Example 2: monitoring the life of a process

In the example above, the process is already started and waiting for a user input. In that situation, it is easy to break into the VM and place our breakpoints at any time.

Often, we will want to break at the start of a process in order to control its behavior as early as possible. WinDbg usually provides exceptions for this set with the sx* commands. However we cannot use these because the kernel is not aware that we are debugging it — this is a trade-off of using the bridge instead of a regular serial connection. See the 2.11.0 documentation for more information about the perimeter.

Thankfully, we can workaround this and monitor a process’ life manually.

Breaking on process start

Let’s try to get into the entry point of a binary. We’ll use hostname as an example, but of course the same principle can be applied on any binary.

A very generic breaking point that will catch every process creation is nt!NtCreateUserProcess:

bp nt!NtCreateUserProcess
g

We now head over to the VM & start the process:

Process start

The VM breaks, and we must let the process creation finish:

Breakpoint 0 hit
nt!NtCreateUserProcess:
fffff800`0bc42290 4055            push    rbp

kd> pt
nt!NtCreateUserProcess+0xad4:
fffff800`0bc42d64 c3              ret

kd> dx @$curprocess
@$curprocess                 : cmd.exe [Switch To]
    KernelObject     [Type: _EPROCESS]
    Name             : cmd.exe
    Id               : 0x8e4
    Handle           : 0xf0f0f0f0
    Threads         
    Modules         
    Environment     
    Devices         
    Io       

As you can see, at that point we are not yet into the context of our new process, even after exiting the function. However this process already exists and its entry point can be found by using the Debugger Objects.

First, we locate the process as we did before:

dx @$targetP = @$cursession.Processes.Where(p => p.Name.ToLower().Contains("hostname")).First()
dx @$targetP.SwitchTo()
.reload /user

NOTE: we could also have checked the handle created by NtCreateUserProcess to find the process.

Now let’s find the entry point and get there:

kd> dx @$targetM = @$curprocess.Modules.First();
@$targetM = @$curprocess.Modules.First()                 : \Windows\System32\HOSTNAME.EXE
    BaseAddress      : 0x7ff70ec20000
    Name             : \Windows\System32\HOSTNAME.EXE
    Size             : 0x9000
    Contents        
kd> dx Debugger.Utility.Control.ExecuteCommand("g " + (@$targetM.Contents.Headers.OptionalHeader.AddressOfEntryPoint + @$targetM.BaseAddress).ToDisplayString())

And we are now at the start of our process:

Break at process start

At this point we can have a precise control over the early life of our process.

Example 3: multi-process recording conditions

WinDbg being connected in kernel mode implies that following the execution between threads or processes is straightforward. Let’s take the example of a program that injects code into a separate, existing process — similar to what malwares sometimes do. We’ll assume we want to skip all the setup prior to the injection and record the injected code only.

First, we start the program and locate its process as we did before. In our example, its kernel object is at 0xffffc8016f587580. Once that is done, we monitor this process for a set of tell-tale calls, so that we can monitor its activity and be certain of what we record:

bp /p ffffc8016f587580 kernelbase!OpenProcess
bp /p ffffc8016f587580 kernelbase!WriteProcessMemory
bp /p ffffc8016f587580 kernelbase!CreateRemoteThreadEx

First, we get the OpenProcess call, telling us that the target for the injection is explorer.exe:

Breakpoint 1 hit
KernelBase!OpenProcess:
0033:00007ffc`ef9972c0 4c8bdc          mov     r11,rsp
kd> r r8
r8=0000000000000584
kd> !process 584 0
Searching for Process with Cid == 584
PROCESS ffffc8016f5ee380
    SessionId: 1  Cid: 0584    Peb: 00b1d000  ParentCid: 0538
    DirBase: 680fc000  ObjectTable: ffff9e848bafdc00  HandleCount: 1580.
    Image: explorer.exe

Let’s see which handle value will be used for this target:

kd> pt
KernelBase!OpenProcess+0x63:
0033:00007ffc`ef997323 c3              ret
kd> r rax
rax=00000000000000ac

Then, we see a memory write onto the target process via the handle previously created:

Breakpoint 2 hit
KernelBase!WriteProcessMemory:
0033:00007ffc`ef9e0b30 488bc4          mov     rax,rsp
kd> r rcx
rcx=00000000000000ac

Finally, the CreateRemoteThreadEx against the target process gives us the injected code’s entry point in r9:

Breakpoint 3 hit
KernelBase!CreateRemoteThreadEx:
0033:00007ffc`ef9d6690 4c8bdc          mov     r11,rsp
kd> r rcx
rcx=00000000000000ac
kd> r r9
r9=00000000048611c2

Now we just have to instruct WinDbg to resume until that code location is hit. It will be in a different process, but that is completely transparent to us:

kd> g 00000000048611c2
0033:00000000`048611c2 e919080000      jmp     00000000`048619e0
kd> !process -1 0
PROCESS ffffc8016f5ee380
    SessionId: 1  Cid: 0584    Peb: 00b1d000  ParentCid: 0538
    DirBase: 680fc000  ObjectTable: ffff9e848bafdc00  HandleCount: 1530.
    Image: explorer.exe

And as we did in previous examples we can now:

  • Head over to the Project Manager
  • Start recording while the VM is paused
  • Resume the VM with g — the recording effectively starts.

As a result, we get a recording that leaves nothing to chance: we know it contains the injected code because we closely monitored the activity of the process prior to recording, and it starts at exactly the moment we wanted.

Supported perimeter

Due to the nature of the bridge connection, only a subset of WinDbg’s feature set is supported. Similarly to connecting WinDbg to a trace, reading memory or registers is supported, as well as code breakpoints.

However, memory breakpoints and any operations that would end up writing to VM are unsupported. This also means the following commands do not work:

  • .process /i to switch to new process (but as we have seen, we can work around that)
  • sx* commands

What about WinTTD?

WinDbg-assisted recording is strictly an improvement over the existing recording features of REVEN. However, this does raise the question: when should we use REVEN instead of WinTTD?

Both tools offer different features and trade-offs. First of all, while the feature sets have a lot in common they vary in significant ways: notably, REVEN provides a tainting engine that is critical in solving certain situations. When it comes to usability, WinTTD is of course directly integrated in WinDbg and very light and easy to start using — on the other hand, REVEN requires setting up virtual machines first. But the reason for this is that the perimeters are different: WinTTD is user-mode only, and REVEN allows the user to record everything including multiple processes, driver or kernel code.

In the end, choosing one or the other will depend on what your use case requires.

Conclusion

In this article we have covered a few examples of situations where using WinDbg on the VM we want to record makes the recording process easier as it allows for far more precision and control, hence much lighter scenarios and smaller analysis lifecycles.

Once again, for more details take a look at the documentation about the WinDbg integration in REVEN.

We hope you will enjoy this latest feature.

Next post: Announcing Tetrane acquisition by eShard
Previous post: Announcing REVEN version 2.11