Our blog is moving to eShard

Tue, 26 Jul 2022 00:00:00 +0000

Following the acquisition of Tetrane by eShard, new articles will be published on the eShard blog starting today.

This post will only provide links to the newly published articles (you can still find all the pre-existing content on the blog home):

Retrieve the drive letter from a device name on Windows

Announcing Tetrane acquisition by eShard

Tue, 26 Jul 2022 00:00:00 +0000

I am really excited to announce that eShard has acquired Tetrane.

At Tetrane, we have spent years rethinking and reinventing how timeless analytics can contribute to vulnerability research. It’s been a crazy, exciting, and fun journey, during which we’ve built an incredible team of very talented people. This is the beginning of another great step forward.

eShard is a leading specialist willing to make cybersecurity tools scalable focusing on security solutions for in-depth security in ICs, software and IoT devices. Working in defense and aerospace, finance, high-tech, health and semiconductor industries, the company helps its customers to address the complex security challenges.

Look at eShard products: Side Channel Analysis, Fault Injection, Firmware Security analysis, Mobile Application Security Testing, and you will clearly understand that this is the perfect fit for REVEN and the team. By combining capabilities and expertise, we will be able to execute our exciting roadmap to offer an even better REVEN to our customers.

Under the eShard brand, REVEN will become a reference tool for vulnerability research in different platforms, including mobile and IoT. In this strategy, the development of know-how material in advanced reverse engineering will help our customers to achieve their missions.

We are all sharing a vision of where we’d like to be and great ambitions for REVEN. Exciting time ahead!

Watch the video of the annoucement below
Go to www.eShard.com
Contact us at reven@eshard.com

Accessing REVEN packages, support, etc.

All REVEN ressources will stay under the Tetrane brand for the next few months:
- Support : & helpdesk
- Documentation
- GitHub
- REVEN install package download links stay the same (see your license email)

Sales & contact

License renewal:
- REVEN Enterprise licenses renewals will be managed under the eShard name.
- REVEN Professional licenses acquisition and renewal stay the same and be mainly online .
- REVEN Free license - get it online.
- Contact the team

Fine-tuned Windows scenarios: debugger-assisted recording with WinDbg

Thu, 16 Jun 2022 00:00:00 +0000

In this article we present a new way to leverage WinDbg to interactively control the VM during the recording of a scenario: this enables the user to more intuitively get to the point of interest and check conditions to control the execution, resulting in a more precise recording.

You can also check the documentation about the WinDbg integration in REVEN in general.

The situation

Making recordings shorter is a key skill when using REVEN. Among the tools and techniques available so far were the automatic binary recording and the ASM stub, which enable introspection and automation during this preliminary phase. In a previous article, we presented an interactive method to control the state of the VM, and thus the recording, from the hypervisor side using QEMU’s native GDB-server mode.

The latter is very versatile and is our go-to method in a number of different situations. However, since neither QEMU nor GDB are aware of the nature of the running kernel, it is also cumbersome: the absence of symbols requires creative workarounds, such as recording and replaying extremely short traces along the way to fall back on REVEN’s OS Specific Information (OSSI).

On the Windows side this led us to look at WinDbg, which can connect to a distant machine or VM and propose a native kernel debugging environment. Unfortunately, it does not seem to be possible to use the modern, fast Ethernet debugger connection with QEMU’s emulated hardware. Falling back on legacy serial debugger connection is possible, but our testing shows it is uncomfortably slow.

As a result we extended the Virtual Machine Instrospection (VMI) tools that powered the automatic binary recording to allow the REVEN-Kd Bridge to connect to a live running VM (whereas the existing WinDbg integration connects to a trace).

Example 1: recording a browser crash

Let us consider the same example as the GDB article: we want to start a recording on a call to chrome!net::HttpBasicStream::SendRequest and stop it when ntdll!KiUserExceptionDispatch is called (when the browser tab crashes). As we will see, using WinDbg for this task is more straightforward because the symbols are resolved automatically.

Connecting

First, let’s restore a VM snapshot where everything is ready for recording. Then, we can start the WinDbg VMI backend:

Next, we connect the bridge to this backend, as we usually do when working with WinDbg on REVEN traces:

NOTE: even though we seem to connect WinDbg to a serial connection, this is actually passing through a named pipe and a network connection to the server — the displayed baud rate does not apply.

Finally, we connect WinDbg to this bridge. Everything is now setup and ready:

Usage

We can now control the VM through WinDbg. For our use case we must:

Find the chrome process.
Place the entry break point.
Resume the VM.

We will use the following commands (see the 2.11.0 documentation for more details):

dx @$cursession.Processes.Where(p => p.Name.ToLower().Contains("chrome")).First().SwitchTo()
.reload /user
bp chrome!net::HttpBasicStream::SendRequest
g

We now head over the VM and click on the exploit’s link. The VM breaks:

At that point, since the VM is paused, we can prepare the next steps in order:

Disable the existing breakpoint as we don’t need it anymore
Place a breakpoint on the crash handler.

We use the following commands:

bd 0
bp ntdll!KiUserExceptionDispatch

We now start the recording on the Project Manager’s page: the recording will actually start when the VM is resumed.

And we can now resume the VM with g. After a while the browser crashed, and the VM breaks again.

The breakpoint is set on the whole VM, so for good measure we should check that the crashed process is our target:

We are in the expected process. If not, we could have resumed the VM and the recording would have resumed as well. In our case, the target has crashed as expected, so we can stop the recording:

At this point, we can remove all breakpoints again and let the VM continue, again to make sure that our tab has crashed. This will not be part of the recorded trace.

We can now confidently commit the recording. The resulting trace will precisely start & stop where we wanted to, with little effort on our part.

Here is a video of the whole process, starting with a connected WinDbg:

Example 2: monitoring the life of a process

In the example above, the process is already started and waiting for a user input. In that situation, it is easy to break into the VM and place our breakpoints at any time.

Often, we will want to break at the start of a process in order to control its behavior as early as possible. WinDbg usually provides exceptions for this set with the sx* commands. However we cannot use these because the kernel is not aware that we are debugging it — this is a trade-off of using the bridge instead of a regular serial connection. See the 2.11.0 documentation for more information about the perimeter.

Thankfully, we can workaround this and monitor a process’ life manually.

Breaking on process start

Let’s try to get into the entry point of a binary. We’ll use hostname as an example, but of course the same principle can be applied on any binary.

A very generic breaking point that will catch every process creation is nt!NtCreateUserProcess:

bp nt!NtCreateUserProcess
g

We now head over to the VM & start the process:

The VM breaks, and we must let the process creation finish:

Breakpoint 0 hit
nt!NtCreateUserProcess:
fffff800`0bc42290 4055            push    rbp

kd> pt
nt!NtCreateUserProcess+0xad4:
fffff800`0bc42d64 c3              ret

kd> dx @$curprocess
@$curprocess                 : cmd.exe [Switch To]
    KernelObject     [Type: _EPROCESS]
    Name             : cmd.exe
    Id               : 0x8e4
    Handle           : 0xf0f0f0f0
    Threads         
    Modules         
    Environment     
    Devices         
    Io       

As you can see, at that point we are not yet into the context of our new process, even after exiting the function. However this process already exists and its entry point can be found by using the Debugger Objects.

First, we locate the process as we did before:

dx @$targetP = @$cursession.Processes.Where(p => p.Name.ToLower().Contains("hostname")).First()
dx @$targetP.SwitchTo()
.reload /user

NOTE: we could also have checked the handle created by NtCreateUserProcess to find the process.

Now let’s find the entry point and get there:

kd> dx @$targetM = @$curprocess.Modules.First();
@$targetM = @$curprocess.Modules.First()                 : \Windows\System32\HOSTNAME.EXE
    BaseAddress      : 0x7ff70ec20000
    Name             : \Windows\System32\HOSTNAME.EXE
    Size             : 0x9000
    Contents        
kd> dx Debugger.Utility.Control.ExecuteCommand("g " + (@$targetM.Contents.Headers.OptionalHeader.AddressOfEntryPoint + @$targetM.BaseAddress).ToDisplayString())

And we are now at the start of our process:

At this point we can have a precise control over the early life of our process.

Example 3: multi-process recording conditions

WinDbg being connected in kernel mode implies that following the execution between threads or processes is straightforward. Let’s take the example of a program that injects code into a separate, existing process — similar to what malwares sometimes do. We’ll assume we want to skip all the setup prior to the injection and record the injected code only.

First, we start the program and locate its process as we did before. In our example, its kernel object is at 0xffffc8016f587580. Once that is done, we monitor this process for a set of tell-tale calls, so that we can monitor its activity and be certain of what we record:

bp /p ffffc8016f587580 kernelbase!OpenProcess
bp /p ffffc8016f587580 kernelbase!WriteProcessMemory
bp /p ffffc8016f587580 kernelbase!CreateRemoteThreadEx

First, we get the OpenProcess call, telling us that the target for the injection is explorer.exe:

Breakpoint 1 hit
KernelBase!OpenProcess:
0033:00007ffc`ef9972c0 4c8bdc          mov     r11,rsp
kd> r r8
r8=0000000000000584
kd> !process 584 0
Searching for Process with Cid == 584
PROCESS ffffc8016f5ee380
    SessionId: 1  Cid: 0584    Peb: 00b1d000  ParentCid: 0538
    DirBase: 680fc000  ObjectTable: ffff9e848bafdc00  HandleCount: 1580.
    Image: explorer.exe

Let’s see which handle value will be used for this target:

kd> pt
KernelBase!OpenProcess+0x63:
0033:00007ffc`ef997323 c3              ret
kd> r rax
rax=00000000000000ac

Then, we see a memory write onto the target process via the handle previously created:

Breakpoint 2 hit
KernelBase!WriteProcessMemory:
0033:00007ffc`ef9e0b30 488bc4          mov     rax,rsp
kd> r rcx
rcx=00000000000000ac

Finally, the CreateRemoteThreadEx against the target process gives us the injected code’s entry point in r9:

Breakpoint 3 hit
KernelBase!CreateRemoteThreadEx:
0033:00007ffc`ef9d6690 4c8bdc          mov     r11,rsp
kd> r rcx
rcx=00000000000000ac
kd> r r9
r9=00000000048611c2

Now we just have to instruct WinDbg to resume until that code location is hit. It will be in a different process, but that is completely transparent to us:

kd> g 00000000048611c2
0033:00000000`048611c2 e919080000      jmp     00000000`048619e0
kd> !process -1 0
PROCESS ffffc8016f5ee380
    SessionId: 1  Cid: 0584    Peb: 00b1d000  ParentCid: 0538
    DirBase: 680fc000  ObjectTable: ffff9e848bafdc00  HandleCount: 1530.
    Image: explorer.exe

And as we did in previous examples we can now:

Head over to the Project Manager
Start recording while the VM is paused
Resume the VM with g — the recording effectively starts.

As a result, we get a recording that leaves nothing to chance: we know it contains the injected code because we closely monitored the activity of the process prior to recording, and it starts at exactly the moment we wanted.

Supported perimeter

Due to the nature of the bridge connection, only a subset of WinDbg’s feature set is supported. Similarly to connecting WinDbg to a trace, reading memory or registers is supported, as well as code breakpoints.

However, memory breakpoints and any operations that would end up writing to VM are unsupported. This also means the following commands do not work:

.process /i to switch to new process (but as we have seen, we can work around that)
sx* commands

What about WinTTD?

WinDbg-assisted recording is strictly an improvement over the existing recording features of REVEN. However, this does raise the question: when should we use REVEN instead of WinTTD?

Both tools offer different features and trade-offs. First of all, while the feature sets have a lot in common they vary in significant ways: notably, REVEN provides a tainting engine that is critical in solving certain situations. When it comes to usability, WinTTD is of course directly integrated in WinDbg and very light and easy to start using — on the other hand, REVEN requires setting up virtual machines first. But the reason for this is that the perimeters are different: WinTTD is user-mode only, and REVEN allows the user to record everything including multiple processes, driver or kernel code.

In the end, choosing one or the other will depend on what your use case requires.

Conclusion

In this article we have covered a few examples of situations where using WinDbg on the VM we want to record makes the recording process easier as it allows for far more precision and control, hence much lighter scenarios and smaller analysis lifecycles.

Once again, for more details take a look at the documentation about the WinDbg integration in REVEN.

We hope you will enjoy this latest feature.

Announcing REVEN version 2.11

Tue, 31 May 2022 00:00:00 +0000

Tetrane is pleased to announce the release of REVEN Enterprise, REVEN Professional and REVEN Free 2.11.

REVEN is a Timeless Debugging and Analysis (TDnA) Platform designed to go x10 faster & x10 deeper while reverse engineering. Technically, REVEN captures a time slice of a full system execution (CPU, Memory, Hardware events) to provide unique analysis features, such as Memory History or forward/backward data flow Taint, that speed up and scale the reverse engineering process.

REVEN version 2.11 is without contest one of the biggest releases of REVEN, filled to the brim with improvements to the entire workflow, from the recording process to the Analysis Python API.

This article covers these new features and tools and the other important changes introduced in this new release.

Analyze, triage and minimize the output of a fuzzer with the Fuzzing & Triage platform
Use debugger-assisted recording to produce short and focused scenarios
Perform richer analyses with the Python API
And More

Analyze, triage and minimize the output of a fuzzer with the Fuzzing & Triage platform

Fuzzing is a very central part of today’s security landscape, as it is a very efficient way to discover crashes that can lead to denial of service, and often to exploitable vulnerabilities.

With its advanced analysis capabilities, REVEN can help with fuzzing too!

This release includes a first iteration of the vision we recently presented. In REVEN Enterprise, the platform allows monitoring the crashes produced by a fuzzer. Each such crash is then automatically recorded, replayed and analyzed, ultimately performing some initial triage of the crashes.

For this first iteration, analysis is supported for Windows scenarios. Depending on the crash, it can automatically find its root cause and link to relevant portions of the input file leading to it.

In all editions, you can use the platform’s analyzer against any existing replayed scenario that was created manually.

Use debugger-assisted recording to produce short and focused scenarios

When working with REVEN, it is important to keep records short and to the point, to avoid increasing the generation time and disk use of replay data. Precise control over when to start and stop recording a scenario greatly helps in that regard.

While the enterprise edition already provides automatic recording facilities since REVEN 2.2, REVEN 2.11 includes a new integration with WinDbg at record time for all editions that enables what we call debugger-assisted recording.

While using debugger-assisted recording, you use the VM as normal to record your scenario. Then, you can break execution of the VM anytime using WinDbg, and use breakpoints on function name, among other WinDbg features. For example, this allows to setup a breakpoint, start the record once it is hit, set a second breakpoint, and stop the record on hitting the second breakpoint.

We plan to release a dedicated article on the topic of debugger-assisted debugging soon.

Perform richer analyses with the Python API

Since its release in REVEN 2.2, the Python API strived to provide reversers with efficient and useful tools for analysis. REVEN 2.11 is an important milestone, with many improvements and some major new features. The following sections present a few highlights.

A convenient API for implementing semantic tainting

Providing a taint API that’s easy to use is a hard problem, in no small part because taint itself is not a simple concept.

Our own implementation of semantic tainting in the Fuzzing Platform and our vulnerability detection scripts inspired us improvements to the API that cover common use cases we encountered time and time again:

Easier way to check if data is tainted. For such a common operation, you can now use the TaintState.is_tainted method to query whether some pieces of data are tainted.

  >>> state = taint.state_at(trace.context_before(1000))
  >>> # Checking if all of a register is tainted
  >>> state.is_tainted(reven2.arch.x64.rax).full
  False
  >>> # Checking if some of a register is tainted.
  >>> not state.is_tainted(reven2.arch.x64.rax).empty
  True
  >>> # Checking if all the requested data is tainted.
  >>> state.is_tainted((reven2.arch.x64.rax, reven2.arch.x64.rbx)).full
  False
  >>> # Checking if some of the requested data is tainted.
  >>> not state.is_tainted((reven2.arch.x64.rax, reven2.arch.x64.rbx)).empty
  True

Remove/Add tainted data and then restart the taint! The TaintState class now provides add/remove methods that allow modifying the state. You can then use the Tainter.taint_from_state method to restart a new taint on the modified state.

  >>> state = taint.state_at(return_from_malloc_ctx)
  >>> # Check if rax is tainted, and remove it from the taint in one swoop
  >>> if state.remove(reven2.arch.x64.rax).empty:
  >>>     # rax wasn't tainted, so the currently tainted data does not depend from this allocation
  >>>     return
  >>> else:  # rax tainted, restart the taint with the modified state
  >>>   taint = tainter.taint_from_state(state)

In semantic tainting, one uses these two facilities to check e.g. that rax is tainted at the end of malloc, and if so restart the taint after removing it (so that the internals of the allocator don’t get tainted).

Introducing: data symbols!

Previously, Binary.symbols allowed to retrieve the function symbols available in a scenario/binary. In REVEN 2.11, this method now also returns the data symbol. This makes it easy to, for instance, locate a module’s global variable and read it.

You can use the new function_symbols/data_symbols sets of methods to filter according to the symbol’s nature.

Directly read structure in memory or in registers

How did we use to read filenames in a NtCreateFile before REVEN 2.11?

>>> import reven2
>>> from reven2 import types
>>> server = reven2.RevenServer("localhost", 13370)
>>> ntoskrnl = next(server.ossi.executed_binaries("ntoskrnl"))
>>> nt_create_file = next(ntoskrnl.symbols("^NtCreateFile$"))
>>> # Finding all files that are created in a call to NtCreateFile
>>> def read_filename(ctx):
...    # filename is stored in a UNICODE_STRING structure,
...    # which is stored inside of an object_attribute structure,
...    # a pointer to which is stored as third argument (r8) to the call
...    object_attribute_addr = ctx.read(reven2.arch.x64.r8, types.USize)
...    # the pointer to the unicode string is stored as third member at offset 0x10 of object_attribute
...    punicode_addr = object_attribute_addr + 0x10
...    unicode_addr = ctx.read(LogicalAddress(punicode_addr), types.USize)
...    # the length is stored as first member of UNICODE_STRING, at offset 0x0
...    unicode_length = ctx.read(LogicalAddress(unicode_addr) + 0, types.U16)
...    # the buffer is stored as third member of UNICODE_STRING, at offset 0x8
...    buffer_addr = ctx.read(LogicalAddress(unicode_addr) + 8, types.USize)
...    filename = ctx.read(LogicalAddress(buffer_addr),
...                           types.CString(encoding=types.Encoding.Utf16, max_size=unicode_length))
...    return filename
...
>>> for (index, ctx) in enumerate(server.trace.search.symbol(nt_create_file)):
...     if index > 5:
...         break
...     print("{}: {}".format(ctx, read_filename(ctx)))
...
Context before #14771105: \??\C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy\resources.pri
Context before #14816618: \??\PhysicalDrive0
Context before #16353064: \??\C:\Users\reven\AppData\Local\...\AC\Microsoft
Context before #16446049: \??\C:\Users\reven\AppData\Local\...\AC\Microsoft\Windows
Context before #16698900: \??\C:\Windows\rescache\_merged\2428212390\2218571205.pri
Context before #26715236: \??\C:\Windows\system32\dps.dll

As you can see it required quite a few comments, as the structure traversal was not very obvious from the code.

In REVEN 2.11, we can make the intent of the code clearer with the newest additions to the types API:

>>> import reven2
>>> from reven2 import types
>>> server = reven2.RevenServer("localhost", 13370)
>>> ntoskrnl = next(server.ossi.executed_binaries("ntoskrnl"))
>>> nt_create_file = next(ntoskrnl.symbols("^NtCreateFile$"))
>>> object_attributes_ty = ntoskrnl.exact_type("_OBJECT_ATTRIBUTES")
>>> # Finding all files that are created in a call to NtCreateFile
>>> def read_filename(ctx):
...    # filename is stored in a UNICODE_STRING structure,
...    # which is stored inside of an object_attribute structure,
...    # a pointer to which is stored as third argument (r8) to the call
...    object_attributes = ctx.read(reven2.arch.x64.r8, types.Pointer(object_attributes_ty))
...    unicode_string = object_attribute.field("ObjectName").deref()
...    length = unicode_string.field("Length").read()
...    return unicode_string.field("Buffer").deref_str(
...        types.CString(encoding=types.Encoding.Utf16, max_size=length)
...    )

You can now directly obtain the instance of the UNICODE_STRING in the OBJECT_ATTRIBUTES structure by dereferencing its ObjectName field. Similarly, you can then obtain the filename buffer and its length by reading the appropriate unicode_string fields, so that you can interpret the buffer as an UTF16 string of characters.

The code is also less “fragile”, as it can now transparently resist a change of layout of the structs it manipulates.

Windows Handle inspection

The new reven2.preview.windows package contains many utilities to help you analyze scenarios targeting Windows. The most significant of these is the ability to inspect the handles manipulated by Windows in the execution trace, as well as the system object they point to.

To do so, you’ll start by wrapping any reven2.trace.Context object inside of a reven2.preview.windows.Context object.

>>> from reven2.preview.windows import Context as WindowsContext
>>> ctx = WindowsContext(ctx)

The newly created ctx behaves like a reven2.trace.Context, but provides additional capabilities that are specific to Windows scenarios. For example, you can use the handles method to list the file handles owned by the current process:

>>> # Focus on process handles, ignore the rest
>>> for handle in ctx.handles(kernel_handles = False, special_handles = False):
...     try:
...         # Request FileObject handles only, otherwise raise exception
...         obj = handle.object(reven2.preview.windows.FileObject)
...     except ValueError:
...         continue
...     print(f"{handle}: {obj.filename_with_device}")
Handle 0x4 (object: lin:0xffffc8016fc2c760): \Device\ConDrv\Reference
Handle 0x44 (object: lin:0xffffc8016fa92810): \Device\HarddiskVolume2\reven
Handle 0x48 (object: lin:0xffffc8016fa94a70): \Device\ConDrv\Connect
Handle 0x50 (object: lin:0xffffc8016fa921d0): \Device\ConDrv\Input
Handle 0x54 (object: lin:0xffffc8016fa91eb0): \Device\ConDrv\Output
Handle 0x58 (object: lin:0xffffc8016fa91eb0): \Device\ConDrv\Output
Handle 0xa4 (object: lin:0xffffc8016f8e3b00): \Device\HarddiskVolume2\reven\output

If you found a handle as an argument to some Windows function (such as NtReadFile), you can also easily find the object it refers to:

>>> handle = ctx.handle(0xa4)
>>> print(handle.object())
File object "\Device\HarddiskVolume2\reven\output" (lin:0xffffc8016f8e3b00)

Analyzing handles makes reversing easier by getting a finer understanding of the data manipulated by the Operating System.

Write more robust scripts with gradual typing

Python has long supported optional type annotations that can be checking by external tools and IDE so as to provide more robust type inference and autocompletion, for the benefit of users.

While some typing has been present in the API for several releases now, REVEN 2.11 reaches a new milestone in typing, with almost 75% of the API now typed (according to mypy). The remaining untyped parts are the most dynamically typed methods, such as reven2.Context.read, tests and implementation details, and a few minor packages (such as the bookmark module). We hope to complete the coverage in a future version of REVEN.

Here’s the read_filename function from above, with the addition of type annotations:

>>> import reven2
>>> from reven2 import types
>>> server : reven2.RevenServer = reven2.RevenServer("localhost", 13370)
>>> ntoskrnl : reven2.ossi.Binary = next(server.ossi.executed_binaries("ntoskrnl"))
>>> nt_create_file : reven2.ossi.FunctionSymbol = next(ntoskrnl.symbols("^NtCreateFile$"))
>>> object_attributes_ty : types.Struct = ntoskrnl.exact_type("_OBJECT_ATTRIBUTES")
>>> # Finding all files that are created in a call to NtCreateFile
>>> def read_filename(ctx: reven2.trace.Context) -> str:
...    # filename is stored in a UNICODE_STRING structure,
...    # which is stored inside of an object_attribute structure,
...    # a pointer to which is stored as third argument (r8) to the call
...    object_attributes : types.StructInstance = ctx.read(reven2.arch.x64.r8, types.Pointer(object_attributes_ty))
...    unicode_string : types.StructInstance = object_attribute.field("ObjectName").deref()
...    length : int = unicode_string.field("Length").read()
...    return unicode_string.field("Buffer").deref_str(
...        types.CString(encoding=types.Encoding.Utf16, max_size=length)
...    )

While these annotations are useful for making your script more robust with tooling, the return type of some expressions, such as ntoskrnl.exact_type("_OBJECT_ATTRIBUTES") or object_attribute.field("ObjectName").deref() is unknown at type checking time, because they rely on reading data in debug objects that are only present at runtime.

To alleviate this issue, the types.StructInstance class provides convenience methods to assert the type of the read or dereferenced value:

>>> import reven2
>>> from reven2 import types
>>> server : reven2.RevenServer = reven2.RevenServer("localhost", 13370)
>>> ntoskrnl : reven2.ossi.Binary = next(server.ossi.executed_binaries("ntoskrnl"))
>>> nt_create_file : reven2.ossi.FunctionSymbol = next(ntoskrnl.symbols("^NtCreateFile$"))
>>> object_attributes_ty : types.Struct = ntoskrnl.exact_type("_OBJECT_ATTRIBUTES")
>>> # Finding all files that are created in a call to NtCreateFile
>>> def read_filename(ctx: reven2.trace.Context) -> str:
...    # filename is stored in a UNICODE_STRING structure,
...    # which is stored inside of an object_attribute structure,
...    # a pointer to which is stored as third argument (r8) to the call
...    object_attributes : types.StructInstance = ctx.read(reven2.arch.x64.r8, types.Pointer(object_attributes_ty))
...    unicode_string = object_attribute.field("ObjectName").deref_struct()  # <- deref_struct indicates we deref to a struct
...    length = unicode_string.field("Length").read_int()  # no need to annotate length with int for mypy or the IDE
...    return unicode_string.field("Buffer").deref_str(
...        types.CString(encoding=types.Encoding.Utf16, max_size=length)
...    )

Using these methods, you don’t need to annotate the result of reading of deref’ing from a struct. These methods are also runtime checked, meaning that if you ever get it wrong, you will get a clean error at the point of the mistake rather than down the line in a confusing way.

More generally, we encourage you to use as much typing as possible in your scripts, and to check them with type checkers like mypy, as this can save a lot of time in debugging for bigger scripts.

And More

New scripts in the library: search the trace for the symbols accessing a given memory range, or conversely for the memory accesses made during the execution of a function symbol.
New “vocabulary” types such as MemoryRange and RegisterSlice.
Made it possible to record intereactions with some USB devices configured as USB passthrough.
Continued performance improvements with the Memory History replay time reduced by 10%.
The hovered transition is now displayed in the timeline.

The full list of improvements and fixes is available in the release notes.

Want to try REVEN? The full power of REVEN is available to test in the Free Edition. For a more guided experience, we also provide an extensive set of learning scenarios online, with tutorials available in most demos.

Interested in REVEN? Compare the features of REVEN Free, Professional and REVEN Enterprise.

Discuss on GitHub!

Keep timeless analysis records to the point with REVEN and GDB

Thu, 28 Apr 2022 00:00:00 +0000

How to get the shortest record for a scenario? Given that a short record is synonymous with less recorded, replayed and analyzed instructions, this is a frequent question we receive about REVEN usage, with the goal of speeding up scenario replay and facilitating subsequent analyses with the Axion GUI and the API.

In the Enterprise Edition, you can use automatic binary recording and ASM stub out of the box from the GUI or with the workflow API to easily take short records, precisely tailored to your needs, possibly as part of an entirely headless process, such as in our upcoming Fuzzing & Triage platform.

This article will demonstrate an approach available in all editions to shorten scenario recordings. It leverages the -s QEMU option that enables you to connect a GDB client to the guest machine, which QEMU exposes as an entire system being debugged. You can then use this GDB client to go to specific code breakpoints and start the record, continue until you reach another code breakpoint or otherwise satisfactory end condition, and then stop the record.

Many of our CVE exploit analyses, such as CVE-2020-15999, had their record shortened using this technique.

Here are the basic steps to follow to perform a GDB-enabled record using REVEN 2.10.2 (also in the Free Edition):

Find the address of the functions you want to add a breakpoint on, in order to derive the start and end of your record.
Start the VM in GDB mode, and attach GDB to the VM.
Use GDB to add breakpoints that pause the VM, and the Project Manager to record your scenario like usual.
Replay your scenario!

Here’s an example of following these steps to record CVE-2020-15999.

1. Finding the addresses of the functions of interest

1.1. Identifying the functions of interest

In CVE-2020-15999, we want to find out what happens when we click a link to an exploit, so a good time to start the record would be when the net::HttpBasicStream::SendRequest method is called in chrome.dll.

From running the exploit, we know that it results in a crash of the Chrome tab where we clicked the link, which means that the process associated with the tab certainly crashed, causing a call to ntdll.dll!KiUserExceptionDispatch. This function seems like a good fit to end the record.

Now that we identified our functions of interest, we need a way to find their actual address in the system that we want to record. A simple way to do this is to record a first, short, scenario that ensures these binaries are executed during the record.

1.2. Preparing the VM

Make sure your VM has been prepared for use with REVEN. Further prepare the VM by installing the vulnerable Chrome, starting a fresh no-kvm session, starting Chrome, accessing the remote directory containing the files for the exploit, and waiting for a while for the CPU of the VM to calm down.

1.3. Important: Taking a live snapshot

When the VM is ready, it is critical that we take a live snapshot with Chrome started, in order to be able to take multiple records with stable addresses. While ensuring stable addresses between records is generally made superfluous by the timeless analysis capabilities of REVEN, using the GDB recording technique actually requires them.

1.4. Make a short scenario executing the target binaries

Our target binaries are chrome.dll and ntdll.dll. The latter has a high chance of being executed in any scenario we’d take, so let’s focus on the former. A good way to ensure it is executed is to click on one of the links displayed in the tab that is connected to the remote directory containing the exploit files. Crucially, we don’t need to wait for the requested page to be loaded after clicking the link, and we should hurry as much as possible to stop the scenario after clicking the link. We can then replay the scenario as normal. To go as fast as possible, one can disable in the replay page the Memory History feature, that is not needed for finding the addresses.

1.5. Finding the addresses of the functions of interest.

After finishing the replay of the scenario, let’s use the Python API to find our addresses of interest:

In [0]: def find_address_of_symbol(server, binary_path, symbol_pattern):
   ...:     binary = next(server.ossi.executed_binaries(binary_path))
   ...:     # Find the first context where the binary has been executed in the trace.
   ...:     # This is why we require a scenario where the target binary has actually been executed.
   ...:     ctx = next(server.trace.search.binary(binary))
   ...:     base_address = ctx.ossi.location().base_address
   ...:     symbol = next(binary.symbols(symbol_pattern))
   ...:     return base_address + symbol.rva
   ...:
In [1]: hex(find_address_of_symbol(server, "chrome.dll", "^net::HttpBasicStream::SendRequest$"))
Out[1]: '0x7fff36d4db50'
In [2]: hex(find_address_of_symbol(server, "system32/ntdll.dll", "^KiUserExceptionDispatch$"))
Out[2]: '0x7fff4f8bfde0'

All done, let’s go to use GDB!

2. Starting the VM in GDB mode, and attaching GDB to the VM

2.1. Starting the VM in GDB mode

Create a new scenario on the same disk snapshot, then on the record page check the “Override custom options” checkbox. In the input field, enter the -s option to enable attaching GDB to the VM.

Then start the VM with the live snapshot you took in step (1.3).

2.2. Starting GDB

Start GDB and attach it to the VM:

; gdb
(gdb) target remote host:1234

The host is the host where the VM is running (if your REVEN installation is on your current machine, it is typically localhost). The port is always 1234.

At this point the VM to which you attached GDB should be frozen, waiting for GDB to let it continue.

3. Using GDB to record the scenario

3.1. Starting the record

Add the breakpoint to net::HttpBasicStream::SendRequest in GDB:

(gdb) b *0x7fff36d4db50
Breakpoint 1 at 0x7fff36d4db50

Then call the continue GDB command to let the VM run until it hits that breakpoint:

(gdb) continue
Continuing.

Lastly, inside of the running VM, click the link to the file that will trigger the exploit.

The VM should freeze again and GDB should report:

Breakpoint 1, 0x00007fff36d4db50 in ?? ()

You can then click the Start button in the Project Manager to start the record. You can take your time, since the VM is stopped, the record will not grow in size.

3.2. Performing the record

Delete your initial breakpoint (so that another call to SendRequest doesn’t freeze the VM again), and add the breakpoint to ntdll.dll!KiUserExceptionDispatch:

(gdb) del br 1
(gdb) b *0x7fff4f8bfde0
Breakpoint 2 at 0x7fff4f8bfde0

Then, simply resume the session in GDB to record the part that interests us:

(gdb) continue
Continuing.

After a short while, you should see:

Breakpoint 2, 0x00007fff4f8bfde0 in ?? ()

There, we reached the second breakpoint. If you want to make sure that ntdll.dll!KiUserExceptionDispatch will be present in the trace, you can add a few more instructions to the record with stepi:

(gdb) stepi
(gdb) stepi
(gdb) stepi

We can now press the Stop and Commit buttons in the Project Manager to finalize our record! Again, since the VM is stopped, there is no need to rush here.

4. Replaying the scenario

You can replay the scenario as normal, excepted that you have to override the custom options in the replay page, in order to remove the -s option that is added here by default since it was present at record time.

If you fail to do so, your replay will fail. If that happens to you, simply delete the few resources that replayed successfully (by using the delete all button) and try again after removing the custom option (the checkbox must be checked and its input field empty).

Conclusion

Here is a small video summarizing the record replay with GDB (after having recovered the addresses of interest):

Applying this technique, we were able to record CVE-2020-15999 in under 230 million instructions, whereas manually controlled records were in the ballpark of the billion instructions at the smallest. It is also significantly easier to perform as we can take our time between each step, without a risk of failing the record.

Of course, using this GDB client has drawbacks when it is not complemented by the automatic recording options provided in the Enterprise edition:

No way to fully automate the workflow using a Python API.
Some of us prefer the interface of other debuggers :-).
More substantially, the GDB client has no access to the OSSI for the emulated VM, and so no knowledge of processes, binaries or symbols inside of the VM. This leads to you having to perform multiple GDB sessions and records to write down the useful addresses beforehand and hoping they remain stable so that you can then put your breakpoints at the relevant addresses. This can be made somewhat easier by using live snapshots, like we did in this walkthrough.

We plan to ship some ergonomic improvements to make this GDB connection more visible and easier to use in a future version of REVEN, and we also have a bigger project involving the ability to use a fully informed debugger to drive the record, so stay tuned ;-)!

A tour of the Rust and C++ interoperability ecosystem

Tue, 22 Mar 2022 00:00:00 +0000

Rust is a programming language with a very interesting value proposition when coming from C++, and so it is only natural to see increasing usage of it in REVEN’s codebase.

We have some internal tooling using the language, the frontend for our Windbg integration is written in Rust, and REVEN version 2.9 introduced Rust in our backend.

Integrating Rust inside of a C++ codebase does present its challenges though: while both languages can talk to the other through the C lingua franca, this common language is poor in abstractions and fraught with perils. Both C++ and Rust provide similar higher-level abstractions and facilities, such as destructors, generic types, member functions, standard strings, standard collections or iterators.

In this article, I will present two practical use cases of C++ <-> Rust interoperability, and some of the tooling that’s available in the ecosystem:

Binding a C++ library to make it available to Rust (C++ -> Rust interop). I made experimental bindings for several of our open-source libraries, such as rvnmetadata and rvnsqlite. These were made in order to experiment with what’s available in the interop ecosystem and assess its viability.
Integrating a Rust crate inside of our C++ backend (Rust -> C++ interop). This direction seems much rarer from reading online discussions, which is surprising to me because Rust has an interesting ecosystem of crates and it frequently makes sense to tap into it. We integrated the pdb and symbolic crates (we were even able to contribute back a bit to the latter, and to its dependencies ).

C++ -> Rust: binding C++ libraries

In the following sections, we’ll see four possible approaches to bind C++ in Rust, each with their pros and cons.

bindgen: automagic bindings

With more than 250 contributors and 100 releases, bindgen is the oldest and most used C++ <-> Rust interop tool and lives inside of the rust-lang organization on GitHub. It is led by Emilio Cobos Álvarez who works at Mozilla on Gecko and Servo.

bindgen automatically generates Rust FFI bindings by parsing C or C++ headers with libclang, and yes it can actually do C++ in some capacity too! When binding C libraries, bindgen is certainly the one tool you will need!

For C++ though, my experience is a bit more mixed, due to several causes:

bindgen is unable to bind large swaths of the standard library (including eg std::string ).
bindgen does not attempt to translate high-level C++ concepts to high-level Rust concepts: it doesn’t handle exceptions, nor a lot of the templates or automatically calling destructors.
bindgen has some hard to fix soundness issues .

Still, when bindgen works for you, it feels really 🌈magical🌈: it generates the full Rust API from the C++ headers, complete with code comments and all enum values. bindgen is rather unique in that it is able to do so without you having to write any kind of glue code.

Here’s how you’d generate the bindings for rvnsqlite using bindgen:

// build.rs
use bindgen;
use cmake;
use std::{env, path::PathBuf};

fn main() {
    // Builds the project in the directory located in `rvnsqlite`, installing it
    // into $OUT_DIR
    let mut config = cmake::Config::new("rvnsqlite");
    // This prevents removal of some functions we'd like to bind
    let dst = config.cxxflag("-fkeep-inline-functions").build();

    println!("cargo:rustc-link-search=native={}/lib", dst.display());
    println!("cargo:rustc-link-lib=static=rvnsqlite");
    println!("cargo:rustc-link-lib=stdc++");
    println!("cargo:rustc-link-lib=sqlite3");

    // Tell cargo to invalidate the built crate whenever the wrapper changes
    println!("cargo:rerun-if-changed=wrapper.h");

    // The bindgen::Builder is the main entry point
    // to bindgen, and lets you build up options for
    // the resulting bindings.
    let bindings = bindgen::Builder::default()
        // The input header we would like to generate
        // bindings for.
        .header("wrapper.h")
        .enable_cxx_namespaces()
        // enable C++
        .clang_args(&["-x", "c++", "--std=c++14", "-fkeep-inline-functions"])
        .opaque_type("std::.*")
        .generate_inline_functions(true)
        .whitelist_type("reven::sqlite::ResourceDatabase")
        // Tell cargo to invalidate the built crate whenever any of the
        // included header files changed.
        .parse_callbacks(Box::new(bindgen::CargoCallbacks))
        // Finish the builder and generate the bindings.
        .generate()
        // Unwrap the Result and panic on failure.
        .expect("Unable to generate bindings");

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("Couldn't write bindings!");
}

// src/lib.rs
#![allow(non_upper_case_globals)]
#![allow(non_camel_case_types)]
#![allow(non_snake_case)]

// Re-export the generated bindings
include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

// Allows bindgen to find the relevant headers we want to generate bindings for.
#include "rvnsqlite/include/sqlite.h"
#include "rvnsqlite/include/resource_database.h"

And a screenshot of the resulting generated documentation.

(note that the \brief commands come from doxygen commands in the source C++ code documentation)

As you can see from the numerous ⚠️ warning signs beside function names, the various generated functions are all unsafe to call. After all, bindgen is generating extern functions from a header and hoping to find the same functions with the correct signatures in the linked C++ library. Therefore calling this function has to be unsafe (the fact that C++ has no language-level distinction between safe and unsafe code is a second reason).

What you cannot see from these screenshots is that the C++ string type used by bindgen is actually [u64; 4], which, while not false (once you gloss other the relocation issues), is not especially that convenient either.

cpp!

cpp! is a macro from the cpp crate, that allows to write C++ code inline into your Rust code. Sounds crazy? It’s actually very useable! Primarily developed by Nika Layzell (mystor) and Olivier Goffart (ogoffart), this crate is used extensively in e.g. qmetaobject (a crate to build Qt/QML applications with Rust).

Compared with bindgen, cpp! both provides more control over the generation, and model higher-level C++ concepts: for example, the companion macro cpp_class! allows to wrap a C++ class with a copy constructor and a destructor in a Rust struct that accordingly implements Clone and Drop. This remains finicky, because that C++ class must be relocatable, e.g. it must be movable in memory using memcpy. Failing that, the generated code will cause soundness issues (the fundamental mismatch here is that in Rust, all value types are movable with a simple memcpy). Notably this disallows binding std::string that contains a small-string optimization in many implementations.

The drawback of having more control is that one has to write “glue-code”, mostly in the form of cpp_class! to wrap C++ types (often adding std::unique_ptr in the process). Furthermore, the lack of facilities to handle e.g. strings mean that we are often juggling with CString and the result is very unsafe. The experience of writing inline C++ inside Rust code is very empowering, but I would still recommend this crate to developers who know both C++ and Rust very well, and didn’t miss their coffee this morning.

(👀 if you got addicted to writing another language inline in Rust source files, Mara Bos released the inline-python crate, that does exactly what is written on the tin. Use at your sole discretion.)

Here is what it looks like to bind some parts of rvnsqlite using cpp!:

// build.rs: this part doesn't change a lot, the bindgen generator is replaced by cpp_build,
// and we are expecting to find the native libraries already installed (with their headers) in TETRANE_OUT_DIR
use std::path::PathBuf;

fn link_rvnsqlite(root_dir: &str) {
    println!("cargo:rustc-link-search=native={}/lib", root_dir);
    println!("cargo:rustc-link-lib=static=rvnsqlite");
    println!("cargo:rustc-link-lib=stdc++");
    println!("cargo:rustc-link-lib=sqlite3");
}

fn link_rvnbinresource(root_dir: &str) {
    println!("cargo:rustc-link-search=native={}/lib", root_dir);
    println!("cargo:rustc-link-lib=static=rvnbinresource");
    println!("cargo:rustc-link-lib=stdc++");
    println!("cargo:rustc-link-lib=boost_system");
    println!("cargo:rustc-link-lib=boost_filesystem");
}

fn link_rvnjsonresource(root_dir: &str) {
    println!("cargo:rustc-link-search=native={}/lib", root_dir);
    println!("cargo:rustc-link-lib=static=rvnjsonresource");
    println!("cargo:rustc-link-lib=stdc++");
    println!("cargo:rustc-link-lib=boost_system");
    println!("cargo:rustc-link-lib=boost_filesystem");
}

fn link_rvnmetadata(root_dir: &str) {
    link_rvnsqlite(root_dir);
    link_rvnbinresource(root_dir);
    link_rvnjsonresource(root_dir);

    println!("cargo:rustc-link-search=native={}/lib", root_dir);
    println!("cargo:rustc-link-lib=static=rvnmetadata");
    println!("cargo:rustc-link-lib=stdc++");
    println!("cargo:rustc-link-lib=pthread");
    println!("cargo:rustc-link-lib=boost_system");
    println!("cargo:rustc-link-lib=boost_filesystem");
    println!("cargo:rustc-link-lib=magic");
}

fn generate_rvnsqlite_bindings(out_dir: &str) {
    let path: PathBuf = PathBuf::from(out_dir);
    let path = path.join("include");

    cpp_build::Config::new()
        .flag("-std=c++14")
        .include(path)
        .build("src/lib.rs");
    println!("cargo:rerun-if-changed=src/lib.rs");
}

fn main() {
    let path = std::env::var("TETRANE_OUT_DIR")
        .expect("Please set TETRANE_OUT_DIR env variable to output dir of an octopus build");
    link_rvnmetadata(&path);
    generate_rvnsqlite_bindings(&path);
}

// src/lib.rs
use std::{
    ffi::CStr, // we will need to juggle these a lot
};

use cpp::{cpp, cpp_class};

// start by including the C++ we'll need
cpp! { {
    #include <rvnmetadata/metadata.h>
    #include <rvnsqlite/resource_database.h>
    #include <experimental/string_view>
    #include <memory>
} }

// define a C++ struct that's essentially a unique_ptr of our opaque struct.
cpp! { {
    struct BoxResourceDatabase {
        using Ptr = std::unique_ptr<reven::sqlite::ResourceDatabase>;
        Ptr value;

        BoxResourceDatabase(Ptr value_) : value(std::move(value_)) {}
    };
} }

// wrap the struct in a Rust struct. The `as` is used to specify its C++ name (declared in the struct above)
// this struct will automatically implement Drop to call the destructor of `BoxResourceDatabase`
cpp_class!(unsafe struct BoxResourceDatabase as "BoxResourceDatabase");

pub struct ResourceDatabase(BoxResourceDatabase);

#[derive(Debug)]
pub enum OpenMode {
    ReadOnly,
    ReadWrite,
}

impl ResourceDatabase {
    pub fn open(filename: &CStr, mode: OpenMode) -> Self {
        let filename = filename.as_ptr();
        let read_only = matches!(mode, OpenMode::ReadOnly);
        // Call the foreign C++ functions.
        // Because these functions exist in the included headers above and will be linked "natively" using a C++
        // compiler, there is no risk of mismatching signature like with bindgen.
        // It is still unsafe because obviously bad stuff can happen (👀 like dereferencing a null ptr...)
        let box_db = cpp!(unsafe [filename as "const char*", read_only as "bool"] -> BoxResourceDatabase as "BoxResourceDatabase" {
            try {
                return std::make_unique<reven::sqlite::ResourceDatabase>(reven::sqlite::ResourceDatabase::open(filename, read_only));
            } catch (reven::sqlite::DatabaseError&) {
                return {nullptr};
            } catch (reven::sqlite::ReadMetadataError&) {
                return {nullptr};
            }
        });
        // FIXME: check nullity of ptr, error handling
        Self(box_db)
    }

    pub fn create(filename: &CStr, metadata: Metadata) -> Self {
        let filename = filename.as_ptr();
        let metadata = metadata.as_opaque_metadata();
        let box_db = cpp!(unsafe [filename as "const char*", metadata as "const reven::metadata::Metadata*"] -> BoxResourceDatabase as "BoxResourceDatabase" {
            try {
                return std::make_unique<reven::sqlite::ResourceDatabase>(reven::sqlite::ResourceDatabase::create(filename, metadata->to_sqlite_raw_metadata()));
            } catch (reven::sqlite::DatabaseError&) {
                return {nullptr};
            } catch (reven::sqlite::WriteMetadataError&) {
                return {nullptr};
            }
        });
        // FIXME: check nullity of ptr, error handling
        Self(box_db)
    }

    /* ... [do useful stuff with DB] ...*/
}

cxx

Last, but not least, cxx is a crate by David Tolnay, of serde and syn fame¹. It proposes a different approach to C++ <-> Rust interoperability, where instead of parsing C++ headers or making you write inline C++, it makes you write a specific Rust module called a “bridge” where you declare the interface (common types and functions). Here is an example of such a bridge for rvnsqlite:

#[cxx::bridge]
mod ffi {
    #[namespace = "reven::sqlite::ffi"]
    enum OpenDatabaseStatus {
        Ok,
        DatabaseError,
        ReadMetadataError,
    }

    #[namespace = "reven::sqlite::ffi"]
    struct OpenDatabase {
        db: UniquePtr<ResourceDatabase>,
        status: OpenDatabaseStatus,
        error_message: String,
    }

    #[namespace = "reven::sqlite::ffi"]
    struct CreateDatabase {
        db: UniquePtr<ResourceDatabase>,
        status: CreateDatabaseStatus,
        error_message: String,
    }

    #[namespace = "reven::sqlite::ffi"]
    enum CreateDatabaseStatus {
        Ok,
        DatabaseError,
        WriteMetadataError,
    }

    #[namespace = "reven::sqlite::ffi"]
    unsafe extern "C++" {
        include!("rvnsqlite-rs/cxx/sqlite-ffi.h");

        #[namespace = "reven::sqlite"]
        type ResourceDatabase;

        fn open(filename: &str, read_only: bool) -> OpenDatabase;

        #[namespace = "reven::metadata"]
        type Metadata = rvnmetadata_rs::CxxMetadata; // re-exported from the CXX bridge in rvnmetadata-rs

        #[namespace = ""]
        type sqlite3;

        fn create(filename: &str, metadata: &Metadata) -> CreateDatabase;

        fn from_memory(metadata: &Metadata) -> UniquePtr<ResourceDatabase>;

        fn metadata(db: &ResourceDatabase) -> UniquePtr<Metadata>;

        fn get(db: Pin<&mut ResourceDatabase>) -> Pin<&mut sqlite3>;
    }
}

The bridge describes common types, declared in Rust and made available to C++, and has an extern C++ section that describes opaque types (Rust will need to use them behind a UniquePtr) and functions to be implemented in C++. Note how the unsafe is being confined to the extern "C++" section. This is because while calling C++ is unsafe (since C++ has no language-level concept of safe/unsafe), at least the signatures are verified by cxx. The choice to not litter each call to C++ functions through the bridge reduces the unsafe noise created by bindgen. You can still mark individual functions as unsafe if they have preconditions that must be met for soundness.

Once written, the bridge is parsed by a proc macro (the #[cxx::bridge] bit) that checks that there are compatible C++ definitions in the declared header (specified by the include!("rvnsqlite-rs/cxx/sqlite-ffi.h"); bit).

The strength of cxx is that many “higher-level” types can be used in the bridge: C++’s std::string (although it cannot be passed by value), std::unique_ptr<T>, std::vector<T>, … Even Rust types such as String, Box or &str are made available to C++ through the bridge.

This makes writing the bridge a very high-level affair, without the CString juggling incurred by the alternatives.

On the other hand, this sort of forces one to write some C++ glue, to accommodate for cxx’s set of types, that is both restricted (e.g. no std::string by value) and expanded (access to some Rust types).

Here is what this glue code is looking like for rvnsqlite:

// sqlite-ffi.h
#pragma once

#include <cstdint>
#include <memory>
#include <rust/cxx.h>
#include <rvnmetadata/metadata-common.h>
#include <rvnmetadata/metadata-sql.h>
#include <rvnsqlite/resource_database.h>

namespace reven {
namespace sqlite {
namespace ffi {
struct OpenDatabase;
struct CreateDatabase;

OpenDatabase open(rust::Str filename, bool read_only);

CreateDatabase create(rust::Str filename, const metadata::Metadata &metadata);

std::unique_ptr<ResourceDatabase>
from_memory(const metadata::Metadata &metadata);

std::unique_ptr<metadata::Metadata> metadata(const ResourceDatabase &db);

sqlite3 &get(ResourceDatabase &db);

} // namespace ffi
} // namespace sqlite
} // namespace reven

// sqlite-ffi.cpp
#include <memory>
#include <rvnsqlite-rs/cxx/sqlite-ffi.h>
#include <rvnsqlite-rs/src/ffi.rs.h>

namespace reven {
namespace sqlite {
namespace ffi {
OpenDatabase open(rust::Str filename, bool read_only) {
  auto cxx_filename = std::string(filename);
  try {
    auto db = std::make_unique<ResourceDatabase>(
        ResourceDatabase::open(cxx_filename.c_str(), read_only));
    return OpenDatabase{std::move(db), OpenDatabaseStatus::Ok, {}};
  } catch (DatabaseError &e) {
    return OpenDatabase{nullptr, OpenDatabaseStatus::DatabaseError, e.what()};
  } catch (ReadMetadataError &e) {
    return OpenDatabase{nullptr, OpenDatabaseStatus::ReadMetadataError,
                        e.what()};
  }
}

CreateDatabase create(rust::Str filename,
                      const reven::metadata::Metadata &metadata) {
  auto cxx_filename = std::string(filename);
  try {
    auto db = std::make_unique<ResourceDatabase>(ResourceDatabase::create(
        cxx_filename.c_str(), metadata::to_sqlite_raw_metadata(metadata)));
    return CreateDatabase{std::move(db), CreateDatabaseStatus::Ok, {}};
  } catch (DatabaseError &e) {
    return CreateDatabase{nullptr, CreateDatabaseStatus::DatabaseError,
                          e.what()};
  } catch (WriteMetadataError &e) {
    return CreateDatabase{nullptr, CreateDatabaseStatus::WriteMetadataError,
                          e.what()};
  }
}

std::unique_ptr<ResourceDatabase>
from_memory(const metadata::Metadata &metadata) {
  return std::make_unique<ResourceDatabase>(
      ResourceDatabase::from_memory(to_sqlite_raw_metadata(metadata)));
}

std::unique_ptr<metadata::Metadata> metadata(const ResourceDatabase &db) {
  return std::make_unique<metadata::Metadata>(
      metadata::from_raw_metadata(db.metadata()));
}

sqlite3 &get(ResourceDatabase &db) {
    return *db.get();
}


} // namespace ffi
} // namespace sqlite
} // namespace reven

One can notice how the Rust types are being used, and how this implements the functions described in the extern "C++" part of the bridge.

In its current state, cxx is still missing a few bits that would be tremendously useful:

The ability to bind Rust enums with payloads, that could be exposed with a std::variant-like API. This ability applied in particular to Option<T>!
Exceptions are mapped to a single error type passed in a Result (or causing a panic, if the function is not declared as returning a result), and the Error variant of a Result is turned into an exception. Again, a std::variant-like API (in both directions) could benefit to have more type-safe exceptions (meanwhile I’m declaring types such as CreateDatabase and OpenDatabase to retrieve errors in a type-safe way).
The concept of iterator, that exists in both C++ and Rust albeit in two very different shapes, is not mapped in any way, although it could be very useful (to be precise, supported types expose iterators in their APIs, but we cannot model it in a cxx bridge).
No support for associated functions.

More generally, there is some kind of dichotomy between the “built-in supported” types and the concepts they enable and the user-defined types that feel less “first-class”.

Still, in practice the approach is very usable. Due to the amount of glue involved, it is, like for cpp!, best to minimize the API surface that will be shared. In contrast with cpp!, cxx bridges C++’s and Rust’s high-level concept, which results in much safer interoperability.

(the build.rs part is very similar to that using cpp, only with cxx instead of cpp)

What about autocxx?

autocxx is a crate built on top of cxx that aims at automating the bridge generation. As such autocxx would retain the high-level-ness of cxx, while binding being automatic like bindgen. The effort is mainly led by Adrian Taylor who works on Chrome security at Google.

Unfortunately, I have not been able to bridge either rvnmetadata or rvnsqlite using autocxx despite several attempts. In the latest attempt, I was able to generate some bridge, but the documentation was missing (apparently due to type alias with private types? Running cargo doc with --document-private-items would fix that issue) and the Metadata type could not be generated (probably due to some unsupported type in its API?).

autocxx is evolving rapidly, so I plan on revisiting it again in some time.

Rust -> C++: Integrating Rust libraries

What about the other direction? The options here are a bit more limited.

cbindgen allows to generate C or C++ headers from the extern "C" fn of a Rust crate, but this approach is very limited:

Although the generated headers can be in C++, there is no support for C++ concepts such as destructors or standard types.
Obviously you need to write the extern "C" fn, which again limits you to repr(C) types.

Fortunately, cxx (and autocxx I guess?) supports an extern "Rust" section in the bridge, which enables to expose opaque Rust types and functions to C++. The proc-macro checks that the items are actually defined in the crate that contains the bridge.

Here is a small example of a cxx for binding the demangling support from the symbolic crate:

/// Types shared with C++.
#[cxx::bridge]
pub mod ffi {
    #[derive(Debug, Clone)]
    #[namespace = "rvnsymbolic"]
    /// Shared struct to replace Option<DemangledSymbol> since this is not currently compatible with CXX.
    ///
    /// When `has_demangled_symbol` is `false`, users of this struct should not read the `demangled_symbol` field.
    pub struct MaybeDemangledSymbol {
        /// The inner demangled symbol, if any.
        pub demangled_symbol: DemangledSymbol,
        /// Whether or not there is an inner demangled symbol.
        pub has_demangled_symbol: bool,
    }

    #[derive(Debug, Clone, Default)]
    #[namespace = "rvnsymbolic"]
    /// Shared struct that represents a demangled symbol
    pub struct DemangledSymbol {
        /// Only the name of symbol
        pub name: String,
        /// The full symbol with name, parameters and return type
        pub full: String,
    }

    #[namespace = "rvnsymbolic"]
    extern "Rust" {
        fn demangle(symbol: &CxxString) -> MaybeDemangledSymbol;
    }
}

use cxx::CxxString;
use symbolic::{
    common::Name,
    demangle::{Demangle, DemangleOptions},
};

use crate::ffi;

impl ffi::MaybeDemangledSymbol {
    fn none() -> Self {
        Self {
            has_demangled_symbol: false,
            demangled_symbol: Default::default(),
        }
    }
}

/// Attempts to demangle a symbol name, returning None if this fails.
pub fn demangle(symbol: &CxxString) -> ffi::MaybeDemangledSymbol {
    let mangled_name = if let Ok(symbol) = symbol.to_str() {
        Name::from(symbol)
    } else {
        // Early return of "none"
        return ffi::MaybeDemangledSymbol::none();
    };
    let name = mangled_name.demangle(DemangleOptions::name_only());

    let name = if let Some(name) = name {
        name
    } else {
        return ffi::MaybeDemangledSymbol::none();
    };

    let full = mangled_name.demangle(DemangleOptions::complete());
    if let Some(full) = full {
        ffi::MaybeDemangledSymbol {
            has_demangled_symbol: true,
            demangled_symbol: ffi::DemangledSymbol { name, full },
        }
    } else {
        ffi::MaybeDemangledSymbol::none()
    }
}

The lack of Option<T> is keenly felt in the Rust -> C++ direction, and so is the lack of iterators:

/// Types shared with C++.
#[cxx::bridge]
pub mod ffi {
    // clippy rightfully doesn't want the explicit lifetimes in the symbol function,
    // but CXX requires them.
    #![allow(clippy::needless_lifetimes)]

    #[derive(Debug, Clone)]
    #[namespace = "rvnsymbolic"]
    /// Shared struct to replace Option<Symbol> since this is not currently compatible with CXX.
    ///
    /// When `has_symbol` is `false`, users of this struct should not read the `symbol` field.
    pub struct MaybeSymbol {
        /// The inner symbol, if any.
        pub symbol: Symbol,
        /// Whether or not there is an inner symbol.
        pub has_symbol: bool,
    }

    #[derive(Debug, Clone, Default)]
    #[namespace = "rvnsymbolic"]
    /// Shared struct that represents the data of a symbol.
    pub struct Symbol {
        /// The name of the symbol.
        pub name: String,
        /// The virtual address of this symbol relative to the start of the binary.
        pub rva: u64,
        /// If true, the symbol denotes a function, otherwise it denotes some piece of data.
        pub is_function: bool,
        /// Whether or not this symbol is public.
        pub is_public: bool,
        /// The name of the section that this symbol is in.
        pub section_name: String,
    }

    #[namespace = "rvnsymbolic"]
    extern "Rust" {
        type Pdb;

        fn open_pdb(path: &CxxString) -> Result<Box<Pdb>>;
        fn guid(&mut self) -> Result<String>;
        fn age(&mut self) -> Result<u32>;

        /// SAFETY: C++ callers must ensure: Lifetime(Pdb) > Lifetime(Iterator)
        unsafe fn symbols<'a>(&'a mut self) -> Result<Box<PdbSymbolIterator<'a>>>;
    }

    #[namespace = "rvnsymbolic"]
    extern "Rust" {
        type PdbSymbolIterator<'a>;

        pub fn next(&mut self) -> Result<MaybeSymbol>;
    }
}

This can then be used from C++:

for (auto maybe_symbol = symbols.next(); maybe_symbol.has_symbol; maybe_symbol = symbols.next()) {
    const rvnsymbolic::Symbol& symbol = *maybe_symbol;
    // access e.g. symbol.name, symbol.rva, ...
}
// we would prefer a for (const auto& symbol: symbols) {} ...

Still this is big step up from cbindgen and a raw C API.

Actually using these Rust bindings from C++ requires a little CMake dance, and since CMake comes in many shades of pain, it quickly became specific to our use of CMake. Here is a good starting point for your own projects, though.

Conclusion

This concludes this tour of the C++ <-> Rust interop options in today’s ecosystem.

Here’s a summary in table form of the various options:

Crate	Glue code	Common types	Destructors	Exceptions	Templates	Relocatable classes	Rust -> C++ direction
bindgen	No glue code	Not supported	Manually called	Not supported	Not supported	Unsound if bound by value	No
cpp	Extensive glue code	unique_ptr & shared_ptr	Supported	Not supported	Not supported	Unsound if bound by value	No
cxx	Somewhat extensive glue code	Yes, many	Supported	Supported	Supported	Sound by preventing their binding by value😅	Yes
autocxx²	Minimal glue code	Yes	Supported	Supported	Supported	Experimental `moveit` integration	Planned?

For the time being, the best suited approach for Tetrane is cxx, but it is interesting to watch closely how this space evolves. autocxx has a lot of interesting promises, and in practice it already makes sense today to mix and match cxx with bindgen³.

Having a good interoperability story with C++ is very important for Rust to be used in C++ codebases, let’s hope that the ecosystem continues to improve!

Footnotes

David Tolnay is responsible of around 1/13 of the total downloads from crates.io . ↩
The author of this article could not get his projects to work with autocxx at the time of writing. ↩
We do this for example to generate the ResourceType enum of our rvnmetadata library with bindgen automatically, then use it in the CXX bridge. ↩

Automatic post-fuzzing triage and automation using REVEN

Tue, 08 Mar 2022 00:00:00 +0000

As those of you lucky enough to attend the great OffensiveCon 2022 might be aware, we are developing a Fuzzing & Triage platform based off REVEN Enterprise Edition.

The platform aims at automating root cause analysis by deriving advanced information about crashes found by a fuzzer, such as the origin of the crash (which bytes from the input files are directly causing the crash), the underlying vulnerability (buffer overflow, use-after-free…) and the coverage information for the crash.

The platform comes with a visualizer to display a report for each crash, with the ability to open the corresponding scenario for analysis with Axion (REVEN’s GUI).

How it works

The way the platform works can be summarized with the following diagram:

The platform can watch any directory for crash files. For each crash file appearing in the watched directory, the executor component of the platform will launch several steps:

Record a fresh REVEN scenario from the test harness + the input file causing the crash. The record uses the workflow API that’s available in the Enterprise Edition of REVEN, to automatically start recording at the beginning of the binary’s execution, and stop when it completes (or crashes, as is generally the case when using inputs from a fuzzer).
Replay the recorded scenario, still using REVEN’s workflow Python API.
Analyze the replayed scenario, using REVEN’s analysis Python API:
1. Find the crash point by searching for ntdll!KiUserExceptionDispatch with the call search. We also extract interesting information about the crash, such as the kind of crash (division by zero, page fault, etc).
2. Find the origin of the data causing the crash, using semantic tainting in the backward direction. What is semantic tainting? It is a layer built on top of REVEN’s taint that allows to find higher-level information about the data flow. More about semantic tainting in the next section.
3. Further minimize the number of “unique crashes” from the initial number reported by AFL, deriving a hash from the backtrace at the crash point and the backtraces of the origin points.

The current status and the final report for each crash file can be monitored live in the visualizer view, a web page locally served by the platform.

Semantic tainting

The taint engine of REVEN is pretty unique in that it can follow the dataflow in the backward direction, essentially answering the question: “Where is this data coming from?”. A detailed introduction to REVEN’s taint can be found in this previous article.

However the core taint engine focusses on giving the low-level information of which registers/memory areas are tainted, and which instructions the dataflow goes through.

To provide a more meaningful origin for crashes, such as the bytes in the input file responsible for the crash, or the allocated buffer causing the crash, we need to augment this low-level information with semantic information.

To achieve this goal, we look for specific, known functions in the backtrace while iterating over the tainted instructions. When a known function is hit, we then look if its return value or arguments are currently tainted. If it is the case, we can ascribe a meaning to that event, depending on what is tainted and on the function.

Here are a few examples:

The return value of malloc is tainted. This means that the tainted data originates with an allocated buffer, of size described by malloc’s parameter.
Part of the Buffer argument to NtReadFile is tainted. This means that the tainted data originates with the file pointed by the FileHandle argument. The precise offsets in the file can be found with some black magic involving reading the kernel structures to retrieve information about the handle.

We applied semantic tainting to a toy library with known vulnerabilities that we used as a fuzzing target, as well as on several crash files for vulnerable versions of clang-format.

Experimenting with Semantic tainting on a toy library

Here are some results from these tests:

On the toy library, we found a divide by zero error:

UID
  d60b2c50563601432424993b1a139546130b209825dc2e27ddc46a09ce975ab6

Type
  DIVIDE_ERROR

Desc
  Division by '0' ('rax')

Transition
  #2094654 divide error while executing div rax

Faulted data origin(s)
  [File at #53733] 'C:\reven\id_00000,sig_08,src_000000,time_0,execs_29,op_havoc, rep_4' at offset(s): [20:20]

Semantic tainting retraces the divide error to the 20th byte in the input file that is the source of the crash.

Another interesting case is this data pagefault:

UID
  44f18979b27dcf4ecf54234da716026aafb7d2ddcf58926653284e9c3d0af81e

Type
  DATA_PAGEFAULT

Desc
  Pagefault when accessing 'ds:0x16922987000' ('[rax]')

Transition
  #4154895 page fault while executing movzx edx, byte ptr ds:[rax]

Faulted data origin(s)
  1. [File at #3321421] 'C:\reven\id_000006,sig_08,src_000000,time_31,execs_75,op_havoc,rep 8' at offset(s): [0:7]
  2. [Allocation at #3320437] ‘malloc(116) = 0x16922981490'

We can see that the value comes both from bytes 0 to 7 of the input file, and from an allocation of size 116 that returned the pointer 0x16922981490. The pagefault was caused by accessing 0x16922987000, which is out of the bounds of the allocated object.

Applying Semantic tainting on `clang-format` crashes out of OSS-Fuzz

Our first clang-format crash was the following:
- source: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44956
- repro on Linux: Debian clang-format version 13.0.1-3 from Debian Bullseye (stable)
- repro on Windows: clang version 12.0.0 from https://llvm.org/builds/
- testcase: https://oss-fuzz.com/download?testcase_id=4821442364047360
- crash type: stack-overflow
- date: Tue, Feb 22, 2022 Running the analysis generated the following report:
```
UID
  None

Type
  DATA_PAGEFAULT

Desc
  DATA PAGEFAULT at #284863280

Transition
  #284863280 page fault while executing call 0x7ffac63ad4f8

Faulted data origin(s)
  Unknown
```
We can see that the crash occurs because of a data pagefault while executing a call, pointing towards a stack overflow.
The second clang-format crash was from OSS-Fuzz too, and generated the following report:
```
UID
  5bf7818a528b310fd9066d22ef1cc1e538b5d43599131a2506d84ece60cdf10f

Type
  DATA_PAGEFAULT

Desc
  Pagefault when accessing 'ds:0x38' ('[rcx+0x38]')

Transition
  #12390412 page fault while executing mov eax, dword ptr [rcx+0x38]

Faulted data origin(s)
  [Code at #12369993] 'mov qword ptr [rax+0xc0], 0x0'
```
This time the data page fault is retraced by the analysis to an instruction where the immediate 0 is written to memory. This means that the page fault is cause by dereferencing a (quasi-)null pointer. The fact that it is an immediate tells us that the user cannot control this value.

Conclusion

The Fuzzing & Triage platform currently in development at Tetrane brings both integration with fuzzing tool chains and advanced automated crash root cause analysis capabilities thanks to a semantic layer on top of REVEN’s taint engine. We have tested it both on a specifically crafted toy library and on recent actual crashes detected by OSS-Fuzz on clang-format.

The first results look promising and adding this semantic layer is an exciting development that we will carry on in the future!

Coming Soon: Join the beta!

Our current goal is to release a first iteration of REVEN’s Fuzzing & Triage platform in the next version of REVEN Enterprise Edition, hardening the platform by testing it on more targets, making it more flexible (such as allowing the user to plug their own analysis and generate customized reports), displaying more information… A beta program is open for current Enterprise customers. Get in touch for early access.

Inspired by what you saw? Feel free to ask any question or give feedback on our Community Github, our Discord channel or Twitter!

Yes, race conditions can be detected with a single core Timeless Debugging and Analysis platform!

Tue, 01 Mar 2022 00:00:00 +0000

As REVEN emulates a single core machine, a frequent question we receive is if it is possible to use REVEN to analyze race conditions. There can be a lot of confusion around what is possible or not, so in this article we hope to address this by giving examples of categories of scenarios that REVEN can or cannot analyze.

Don’t know about REVEN yet? REVEN is a Timeless Debugging and Analysis (TDnA) platform that enables you to record an entire system inside of a VM during a slice of time, and then indexes data about what was recorded and present it to you through a GUI and a Python API, so that you can analyze what happened using the advanced features of REVEN, such as the entire history of memory accesses in the recorded trace, or data flow analysis in both forward and backward direction.

Some categories of race conditions

First, what is a race condition? A race condition is a flaw that occurs when the timing or ordering of events affects a program’s correctness¹.

Basically, REVEN can analyze race conditions that rely on concurrency, not parallelism. Concurrency is when multiple threads (or processes, remember that REVEN records the full system!) compete for running concurrently on the same resource, while parallelism is when multiple threads/processes run simultaneously (because there are eg multiple physical cores to run things at the same time).

A specific kind of race condition is the data race, that occurs when some piece of data is both written to and read from concurrently by unsynchronized threads/processes. Only concurrency, not parallelism, is necessary to observe most data races. However some of them depends on architectural behavior such as per-core memory cache². Such data races will not produce the same results with a single processor since its cache would always be fresh. Anyway, REVEN currently works at the instruction level and does not attempt to emulate processor caches. Such race conditions cannot be analyzed with REVEN.

Furthermore, from C or C++’s point of view, data races invoke undefined behavior (UB)³. As such, the generated code is not bound to the “as if” rule normally enforced by the compiler (because the input program breaks assumptions made by the compiler). Such undefined behavior can manifest in the generated code eliding e.g. some reads of values⁴. This kind of transformations⁵ can be difficult to understand after the fact when analyzing the binary code. When using REVEN, we might be able to highlight some missing data flow connections using REVEN’s taint, but it is unclear that this would suffice for the analysis.

OK, what does this leave us? Plenty, actually. Lots of race conditions depend on several threads or processes having unsynchronized access to the same resource (be it memory, a file or something else) over multiple instructions, in which case concurrency is all your need for observing the race condition.

To recap, here are the possible race conditions:

Race conditions	REVEN-Analyzable?
Stale architectural caches observed by data race	No
UB artifacts caused by data race	Hardly so
Other data races	Yes
Race for other resources	Yes

Analyzing a Chrome data race

As an example of a data race that can be analyzed using REVEN, let’s do the Root Cause Analysis using REVEN of CVE-2021-21166, a data-race-induced buffer overflow in the WebAudio component of Chrome:

We start by recording the CVE. This involves installing a vulnerable version of Chrome in a VM prepared to run with REVEN, uploading the PoC (from Google Project 0) to that VM, checking it reproduces on the test setup under KVM, and lastly recording the crash. Then the scenario was replayed in around 45 minutes on the author’s laptop, for 37GB of data.
Then on to the analysis! From our record (and the framebuffer), we can see that the Chrome tab loading the exploit crashes, so let’s find the crash of the process associated with the crash with a Symbol Call search on ntdll!KiUserExceptionDispatch.
The crash is caused by a page fault when dereferencing rdi that is not mapped, occurring in the std::sort function from the C++ standard library. Let’s use backward tainting on rdi to find out where the faulty value is coming from!
Backward tainting on rdi indicates that it is used as a kind of “current pointer” for sorting. It is ultimately tracked back by the taint to the first argument of std::sort (ie the rcx register), which is the pointer to the beginning of the buffer that needs sorting.
Applying the memory history on the buffer pointed to by rcx indicates that it is written to during the execution of std::sort.
Going to that write, we can see in the GUI’s status bar that it is being performed by a different thread in blink::AudioBus. Because std::sort is not thread-safe, this write is causing the crash. More specifically, the sort expects to find some sentinel value. Here, it is concurrently rewritten, and as a result it overflows the buffer up until the end of the mapped page.

So yes, as demonstrated, it is possible to analyze such a race condition CVE using REVEN on a single core and applying the classic REVEN analysis tools (taint, memory history).

Using the API, we could even attempt to automatically detect writes to data from another thread during the execution of a function reading the same data:

import bisect
import copy
from dataclasses import dataclass

import reven2
from reven2.preview.project_manager import ProjectManager


pm = ProjectManager("http://127.0.0.1:8880")
c = None
c = pm.connect("CVE-2021-21166-Chrome")
server = c.server


crash_ctx = server.trace.context_before(216269596)  # found from Axion, could be found automatically
crash_tr = crash_ctx.transition_before()
call_tr = crash_tr.step_out(is_forward=False)  # go to the beginning of the crashing function
call_thread = call_tr.context_before().ossi.thread().id  # thread that crashed

### Script utils

class MemoryRange:
    """
    Represents a range of memory with insertion, union and address-sort
    """
    def __init__(self, address, size):
        self.address = address
        self.size = size

    def union(self, other):
        left = self.address.offset
        left_end = left + self.size
        right = other.address.offset
        right_end = right + other.size

        start = left if left < right else right
        end = left_end if left_end > right_end else right_end
        if start >= end or ((end - start) > (self.size + other.size)):
            return None

        address = copy.copy(self.address)
        address._offset = start
        return MemoryRange(address, end - start)

    def __str__(self):
        return f"[{self.address} ; {self.size}]"

    def __le__(self, other):
        return self.address <= other.address

    def __lt__(self, other):
        return self.address < other.address

    def intersection(self, other):
        left = self.address.offset
        left_end = left + self.size
        right = other.address.offset
        right_end = right + other.size

        start = left if left > right else right
        end = left_end if left_end < right_end else right_end

        if start >= end:
            return None

        address = copy.copy(self.address)
        address._offset = start
        return MemoryRange(address, end - start)


class MemoryRangeMap:
    """
    A map of non-overlapping MemoryRange to some data.

    On insertion, ranges that would overlap with the inserted value are merged with the inserted value,
    and their data pushed into a list associated with the resulting merged range.
    """
    def __init__(self):
        self.set = []
        self.data = {}

    def insert(self, memory_range, data):
        data = [data]
        if not self.set:
            self.set.append(memory_range)
            self.data[memory_range.address.offset] = data
            return
        index = bisect.bisect_right(self.set, memory_range)
        if index != 0:
            previous = self.set[index - 1]
            union = previous.union(memory_range)
            if union is not None:
                memory_range = union
                del self.set[index - 1]
                previous_data = self.data[previous.address.offset]
                del self.data[previous.address.offset]
                data = previous_data + data
                index -= 1
        while index != len(self.set):
            next = self.set[index]
            union = next.union(memory_range)
            if union is None:
                break
            memory_range = union
            del self.set[index]
            next_data = self.data[next.address.offset]
            del self.data[next.address.offset]
            data = next_data + data
        self.set.insert(index, memory_range)
        self.data[memory_range.address.offset] = data

    def __iter__(self):
        for memory_range in self.set:
            yield (memory_range, self.data[memory_range.address.offset])


@dataclass
class Data:
    """
    Data to insert in our map
    """
    process_name: str
    thread_id : int
    access: reven2.memhist.MemoryAccess

### Script main

call_map = MemoryRangeMap()  # map for accesses made by the thread that crashed
other_map = MemoryRangeMap()  # map for accesses made by other threads/processes
for access in server.trace.memory_accesses(from_transition=call_tr, to_transition=crash_tr):
    ctx = access.transition.context_before()
    thread = ctx.ossi.thread().id
    process_name = ctx.ossi.process().name
    data = Data(process_name, thread, access)
    is_kernel = ctx.read(reven2.arch.x64.cs) & 3 == 0
    if is_kernel:  # ignore kernel accesses that add a lot of noise. Re-enable if necessary
        continue
    if thread == call_thread:
        call_map.insert(MemoryRange(access.physical_address, access.size), data)
    else:
        # ignore reads from other threads that cannot cause synchronization issues to the current thread
        if access.operation == reven2.memhist.MemoryAccessOperation.Write:
            other_map.insert(MemoryRange(access.physical_address, access.size), data)

# display accesses to shared data: such data is read/write from the crashing thread, and was written from another
# thread at least once.
for (m, m_data) in call_map:
    for (n, n_data) in other_map:
        if MemoryRange.intersection(m, n) is not None:
            print(f"{MemoryRange.intersection(m, n)}: {m_data[0].process_name}!{m_data[0].thread_id}: {m_data[0].access} ||| {n_data[0].process_name}!{n_data[0].thread_id}: {n_data[0].access}")

Running this script produces the following output:

[phy:0x392efc00 ; 1024]: chrome.exe!2388: [#214852126 movss xmm1, dword ptr ds:[rdi+0x4]]Read access at @phy:0x392ef000 (virtual address: lin:0x467c00c11000) of size 4 ||| chrome.exe!2528: [#214322911 movups xmmword ptr ds:[rcx], xmm0]Write access at @phy:0x392efc00 (virtual address: lin:0x467c00c11c00) of size 8
[phy:0x5fbbe000 ; 512]: chrome.exe!2388: [#214469528 movss xmm1, dword ptr ds:[rdi+0x4]]Read access at @phy:0x5fbbe000 (virtual address: lin:0x467c00c0a000) of size 4 ||| chrome.exe!2528: [#214348370 movups xmmword ptr ds:[rcx], xmm0]Write access at @phy:0x5fbbe000 (virtual address: lin:0x467c00c0a000) of size 8
[phy:0x607c3c00 ; 1024]: chrome.exe!2388: [#214961340 movss xmm1, dword ptr ds:[rdi+0x4]]Read access at @phy:0x607c3000 (virtual address: lin:0x467c00c13000) of size 4 ||| chrome.exe!2528: [#214323020 movups xmmword ptr ds:[rcx], xmm0]Write access at @phy:0x607c3c00 (virtual address: lin:0x467c00c13c00) of size 8
[phy:0x76ee1000 ; 512]: chrome.exe!2388: [#214217721 movss xmm0, dword ptr ss:[rbp]]Read access at @phy:0x76ee1ffc (virtual address: lin:0x467c00c08ffc) of size 4 ||| chrome.exe!2528: [#214348261 movups xmmword ptr ds:[rcx], xmm0]Write access at @phy:0x76ee1000 (virtual address: lin:0x467c00c08000) of size 8

After around 20 minutes execution time, we automatically found the concurrent write accesses from thread 2528 in Chrome! If applied to a function that is not thread-safe, then it means we have a concurrency error. For thread-safe functions, it should be verified if the accesses are protected by a synchronization primitive.

Conclusion

While deterministic replays in the presence of multiple cores is a difficult problem, likely to require hardware probes⁶, a full-system TDnA like REVEN can be used to analyze a good chunk of race conditions, including those occurring between multiple threads or processes, and including Chrome CVEs. Using REVEN, such race conditions can be automatically analyzed end-to-end in about one hour.

https://blog.regehr.org/archives/490 ↩
https://corensic.wordpress.com/2011/08/15/data-races-at-the-processor-level/ ↩
https://en.cppreference.com/w/c/language/memory_model#Threads_and_data_races ↩
https://www.hboehm.info/c++mm/why_undef.html ↩
On UB allowing compilers to do whatever with your code, here’s a couple of interesting references: https://blog.regehr.org/archives/213, https://raphlinus.github.io/programming/rust/2018/08/17/undefined-behavior.html ↩
“The best hope for general, low-overhead parallel recording seems to be hardware support.” in O’Callahan, R., Jones, C., Froyd, N., Huey, K., Noll, A., & Partush, N. (2017). Engineering record and replay for deployability. In 2017 USENIX Annual Technical Conference (USENIX ATC 17) (pp. 377-389). ↩

REVEN Free Edition - Available as a VM

Tue, 14 Dec 2021 00:00:00 +0000

We are happy to announce that REVEN Free Edition is now also available as a VM for an easy installation (QEMU, Hyper-V, VMWare). Below are the details to follow.

Download

Download REVEN Free Edition download page.

How to configure the VM

The REVEN in a VM installation provides you with a single virtual disk (pre-installed VM disk). This disk must be put in a rightfully configured VM in order to boot properly.

A second empty disk must be created and attached to the VM to store the data.
Nested virtualization must be activated for this VM.
The VM must be accessible over the network.

The second data disk

To store the REVEN data, you will need to attach a second empty disk to the VM. This disk will automatically be formatted and will grow when necessary, so you won’t have any maintenance to do from inside the VM. It must be exposed to the guest as /dev/sdb, so you need to attach it in second position to any controller that would use the SCSI interface in Linux (SCSI, SATA, USB, SAS, Fibre Channel, FireWire, etc…). See this page of the Linux kernel doc for more information.

Nested virtualization

Nested virtualization will allow REVEN to launch hardware accelerated VMs to help you configure them for analysis. This setting is highly dependent on you hypervisor, so we advise you to look at the documentation. Here are some pointers for the most common solutions out there.

Network access

REVEN provides a Web UI to manage your analysis VMs and Scenarios, thus you will need a network access to the VM. This setting is even more dependent on your installation, as your hypervisor, but also your network configuration will come into account. The VM is configured to run DHCP client on any of its interfaces, and display its IPv4 in the MOTD, meaning many configurations that include a DHCP server on your side should work.

Troubleshooting

The VM ends up in emergency mode. There are multiple possible reason for the VM to end up in emergency mode, but the most probable is that it cannot find the second disk. It should be detected in /dev/sdb. Try moving the disk to a different disk controller on your hypervisor, otherwise open an issue.
I can’t access the Web UI even if the VM doesn’t seem to complain about anything. There probably is an issue with your network configuration. To help debug any problem, please note that both ping and ssh will answer and that no firewall has been configured on the VM. If you cannot figure it out, feel free to open an issue.

REVEN Free Edition - Install Party

Tue, 14 Dec 2021 00:00:00 +0000

Join us for the REVEN Free Edition Install Party on December 21 at 10am EST
->

If you are curious about REVEN, its capabilities, and did not yet took time to discover it, it is the perfect occasion for you to install REVEN Free Edition and try it out. You will also have live access to the development team to provide support or answer more general questions about the product.

During the livestream, we will go through the points below and the team will be there to provide live support and for Q&A.

Getting REVEN running on your own system
Recording your first trace
Getting an introduction to the analysis UI & API

Add the party to your agenda (details to join included)

Google Calendar - Outlook Calendar - Office 365 Calendar - Yahoo Calendar - ICS file

Steps to have everything you need at hand for the party

1- Download the REVEN Free Edition that fits you from download page. The Party will mainly focus on the VM option as it should be the easiest option for you see Install the VM.
2- Download a VM of the system you want to record with REVEN:
- the Debian Stretch VM,
- on Microsoft’s web site a Window 7 x86 VMs.
You are all set!

Logistics

The livestream will be broadcast on Tetrane’s YouTube channel.
The chat & Q/A will be on this Discord’s channel.

Our blog is moving to eShard

Announcing Tetrane acquisition by eShard

Accessing REVEN packages, support, etc.

Sales & contact

Fine-tuned Windows scenarios: debugger-assisted recording with WinDbg

The situation

Example 1: recording a browser crash

Connecting

Usage

Example 2: monitoring the life of a process

Breaking on process start

Example 3: multi-process recording conditions

Supported perimeter

What about WinTTD?

Conclusion

Announcing REVEN version 2.11

Contents

Analyze, triage and minimize the output of a fuzzer with the Fuzzing & Triage platform

Use debugger-assisted recording to produce short and focused scenarios

Perform richer analyses with the Python API

A convenient API for implementing semantic tainting

Introducing: data symbols!

Directly read structure in memory or in registers

Windows Handle inspection

Write more robust scripts with gradual typing

And More

Keep timeless analysis records to the point with REVEN and GDB

1. Finding the addresses of the functions of interest

1.1. Identifying the functions of interest

1.2. Preparing the VM

1.3. Important: Taking a live snapshot

1.4. Make a short scenario executing the target binaries

1.5. Finding the addresses of the functions of interest.

2. Starting the VM in GDB mode, and attaching GDB to the VM

2.1. Starting the VM in GDB mode

2.2. Starting GDB

3. Using GDB to record the scenario

3.1. Starting the record

3.2. Performing the record

4. Replaying the scenario

Conclusion

A tour of the Rust and C++ interoperability ecosystem

C++ -> Rust: binding C++ libraries

bindgen: automagic bindings

cpp!

cxx

What about autocxx?

Rust -> C++: Integrating Rust libraries

Conclusion

Footnotes

Automatic post-fuzzing triage and automation using REVEN

How it works

Semantic tainting

Experimenting with Semantic tainting on a toy library

Applying Semantic tainting on clang-format crashes out of OSS-Fuzz

Conclusion

Coming Soon: Join the beta!

Yes, race conditions can be detected with a single core Timeless Debugging and Analysis platform!

Some categories of race conditions

Analyzing a Chrome data race

Conclusion

REVEN Free Edition - Available as a VM

Download

How to configure the VM

The second data disk

Nested virtualization

Network access

Troubleshooting

REVEN Free Edition - Install Party

Add the party to your agenda (details to join included)

Steps to have everything you need at hand for the party

Logistics

Applying Semantic tainting on `clang-format` crashes out of OSS-Fuzz