Address Windowing Extensions: Struck by AWE

Windows internals research provides a vast amount of room to explore, and sometimes you come across a particularly interesting subject you may have not known about or heard of: a subject that has very minimal documentation and seems to be a remnant of the days prior to native 64bit support.

These are the times that excite you because you’re certain you will find that brand new vulnerability that may lead to an exploit that affects all versions of the Windows Operating System.

Although this isn't one of those times for me, I wanted to share this “failure” (non-discovery) with you anyways and with any luck you may find something I missed during my own research. I use the term failure here very loosely because in reality, there is no failure when knowledge is gained. While I did not accomplish what I set out to do, I uncovered a wealth of new knowledge that may help you or I in future projects.

This is the beauty of security research. Knowledge is the ultimate victory.

Discovery

During some loosely related research I came across an API that I had never seen or used before: NtMapUserPhysicalPages. It sounded like an interesting name for an API. My first assumption was that this API allowed a user mode program to manipulate physical memory. After some MSDN-fu that's kind of what it was. This API was part of a technology Microsoft called Address Windowing Extensions (AWE).

From MSDN:

Address Windowing Extensions (AWE) is a set of extensions that allows an application to quickly manipulate physical memory greater than 4GB.

AWE solves this problem by allowing applications to directly address huge amounts of memory while continuing to use 32-bit pointers.

These extensions were created to address memory constraints with the 32-bit memory model. However, this isn't particularly what interested me. The documentation goes on to explain some of the restrictions placed when using this extension. These restrictions are what I was particularly interested in ‘testing’:

Virtual address ranges allocated for the AWE are not sharable with other processes (and therefore not inheritable). In fact, two different AWE virtual addresses within the same process are not allowed to map the same physical page. These restrictions provide fast remapping and cleanup when memory is freed.

At a high level, what this extension allows us to do is reserve physical pages of memory that are non-pageable in order to map them into our user-mode virtual address space. This reservation of non-paged memory allows fast memory access and fast memory remapping among the other stated advantages.

So what API’s are needed to utilize these Extensions?

MSDN lists them as:

  • VirtualAlloc/Ex
  • AllocateUserPhysicalPages
  • MapUserPhysicalPages
  • MapUserPhysicalPagesScatter
  • FreeUserPhysicalPages

Oftentimes the best ideas come from challenging documentation. Documentation of an API or set of functions may declare one thing but the code may contain some flawed implementation of its intent. It’s our duty to challenge these intentions.

In order to actually use this extension, the host Operating System (OS) must have a specific policy enabled for the user (Lock Pages in Memory Policy). This policy is disabled by default for all users in a standard Windows installation; however, some machines running a variant of Windows Server with SQL Server have it enabled.

The Goal

After reading the documentation for AWE I decided to think about some ways we could (ab)use it. Can we somehow allow Process B to remap physical pages allocated by Process A utilizing the fast remapping capabilities of AWE?

It all starts with one question: let’s explore.

Throwing the First Punch (Round One)

We’ve got a basic idea of what we need to work with AWE, so let’s use the extension the way it was intended. MSDN tells us what API’s were needed so a simple usage looks like the code below:

      LPVOID lpvMemoryWindowOne = nullptr;
      LPVOID lpvMEmoryWindowTwo = nullptr;
      SYSTEM_INFO sysInfo = { 0 };

      ULONG_PTR ulNumPages, ulNumOriginalPages = 0;
      ULONG_PTR * ulPfnArray = nullptr;
      ULONG ulPfnArraySize = 0;
      ULONG ulRequestedMemory = 4096 * 5;

      // Ensure correct privileges are set
      if (!LoggedSetLockPagesPrivilege(GetCurrentProcess(), TRUE))
      {
            _tprintf(_T("Failed to set proper privs\n"));
            return 0;
      }

      GetSystemInfo(&sysInfo);

      // Determine number of physical pages we're going to request.
      ulNumPages = ulRequestedMemory / sysInfo.dwPageSize;
      ulPfnArraySize = ulNumPages * sizeof(ULONG_PTR);

      ulPfnArray = (ULONG_PTR*)HeapAlloc(GetProcessHeap(), 0, ulPfnArraySize);

      ulNumOriginalPages = ulNumPages;
      if (!AllocateUserPhysicalPages(GetCurrentProcess(), &ulNumPages, ulPfnArray))
      {
            _tprintf(_T("Failed to Allocate Physical Pages, Error: %X\n"), GetLastError());
            getchar();
            return 0;
      }

      // Create the window
      lpvMemoryWindowOne = VirtualAlloc(NULL, ulRequestedMemory,
            MEM_RESERVE | MEM_PHYSICAL | MEM_TOP_DOWN, PAGE_READWRITE);
      if (lpvMemoryWindowOne == NULL)
      {
            _tprintf(_T("Failed to allocate memory!\n"));
            getchar();
            return 0;
      }

      // Map the physical pages for this window region
      if (!MapUserPhysicalPages(lpvMemoryWindowOne, ulNumPages, ulPfnArray))
      {
            _tprintf(_T("Failed to map physical pages: Error = %X\n"), GetLastError());
            getchar();
            return 0;
      }


The code flow is as follows:

1.     Determine the current system's page size
2.     Determine the amount of physical pages we'll be requesting. (AmountOfMemoryInBytes/SystemPageSize)
3.     Allocate a buffer for the array of page frame numbers we’ll receive from the kernel (HeapAlloc)
4.     Call AllocateUserPhysicalPages(), this API reserves the physical pages for this process and returns the Page Frame Numbers to the caller (why?).
5.     Allocate the virtual address space to be used (VirtualAlloc with MEM_PHYSICAL | MEM_RESERVE only)
6.     Map the physical pages to the virtual address window (MapUserPhysicalPages)

The memory Windows used for this mapping are now guaranteed to be committed in memory and never paged-out. Also, memory regions can extend the official 2GB virtual space provided for 32bit applications if enough physical memory is available.

Testing Documentation (Round Two)

As the documents previously stated: “two different AWE virtual addresses within the same process are not allowed to map the same physical page.” We can easily test this, taking our original code above and just adding another Virtual Window we have this code, which turns out to prove the documentation is correct:

      // Create the first window
      lpvMemoryWindowOne = VirtualAlloc(NULL, ulRequestedMemory,
            MEM_RESERVE | MEM_PHYSICAL, PAGE_READWRITE);
      if (lpvMemoryWindowOne == NULL)
      {
            _tprintf(_T("Failed to allocate memory!\n"));
            getchar();
            goto done;
      }

      // Second Window
      lpvMemoryWindowTwo = VirtualAlloc(NULL, ulRequestedMemory,
            MEM_RESERVE | MEM_PHYSICAL, PAGE_READWRITE);
      if (lpvMemoryWindowTwo == NULL)
      {
            _tprintf(_T("Failed to allocate second window\n"));
            getchar();
            goto done;
      }

      // Map the physical pages for this window region
      if (!MapUserPhysicalPages(lpvMemoryWindowOne, ulNumPages, ulPfnArray))
      {
            _tprintf(_T("Failed to map physical pages: Error = %X\n"), GetLastError());
            getchar();
            goto done;
      }

      // This should fail according to docs
      if (!MapUserPhysicalPages(lpvMemoryWindowTwo, ulNumPages, ulPfnArray))
      {
            _tprintf(_T("Could not map second window!: Error: %X\n"), GetLastError());
            getchar();
            goto done;
      }


Creating two virtual windows and attempting to map them to the same physical pages will in fact fail according to the documentation and this sample code.

Nothing to see here, so let’s move on.

Remapping Physical Pages (Round Three)

AWE allows what it calls fast remapping. Essentially, this allows one to keep data in physical memory as long as needed and if this data needs to be mapped to another virtual address it can easily do so with no performance impact due to the physical pages never being paged out (to disk) or succumbing to trimming. An example of the flow is shown in the below code snippet: (Error checking removed for clarity).

        // Create the window
        lpvMemoryWindowOne = VirtualAlloc(NULL, ulRequestedMemory,
                    MEM_RESERVE | MEM_PHYSICAL, PAGE_READWRITE);

        // Second Window
        lpvMemoryWindowTwo = VirtualAlloc(NULL, ulRequestedMemory,
                    MEM_RESERVE | MEM_PHYSICAL, PAGE_READWRITE);

        // Map the physical pages for this window region
        if (!MapUserPhysicalPages(lpvMemoryWindowOne, ulNumPages, ulPfnArray))
        {
                    _tprintf(_T("Failed to map physical pages: Error = %X\n"), GetLastError());
                    goto done;
        }

// Write some data to the Virtual Window
        if (!WriteData(lpvMemoryWindowOne, ulRequestedMemory))
        {
                    goto done;
        }

        //Unmap the first Window
        if (!MapUserPhysicalPages(lpvMemoryWindowOne, ulNumOriginalPages, NULL))
        {
                    goto done;
        }

        // Now map the physical pages to the new window
        if (!MapUserPhysicalPages(lpvMemoryWindowTwo, ulNumPages, ulPfnArray))
        {
                    goto done;
        }


An initial call to MapUserPhysicalPages is made to map the physical pages to the virtual window; data is written into this window then the Pages are unmapped using a call to MapUserPhysicalPages with a NULL value for the PfnArray parameter. MapUserPhysicalPages is then used to remap those physical pages to a new virtual window. The data is written once and can be mapped to a new virtual address without the penalty that comes with paging in memory from disk. At the end of this sequence, the data written into virtual window one will now reside in virtual window two, and virtual window one will revert back to a RESERVED state.

Remote Remapping (Knockout)

At this point we’ve read documentation on the APIs related to AWE and have a very basic working knowledge of it. The goal was to see if there were any implementation bugs with these extensions regarding the physical pages across process boundaries. The basic idea behind remote remapping is:

  • Process A allocates room for physical pages (AllocateUserPhysicalPages())
  • Process A creates a virtual window A (VirtualAlloc())
  • Process A writes data to virtual window
  • Process A maps the physical pages to the virtual window A (MapUserPhysicalPages)
  • Process A saves the data related to the page frame numbers for use with process B
  • Process A unmaps its virtual window (MapUserPhysicalPages)
  • Process B also allocates physical pages (AllocateUserPhysicalPages) → We need this to happen in order to create an internal structure discussed later.
  • Process B creates a virtual window B
  • Process B attempts to map physical pages from process A to virtual window B

The assumption being made here is that because the documentation states that these pages are not included in the process’s working set, it has to be managed by some other mechanism we don’t know about yet. Furthermore, if the pages are committed, and we control when and how they are released, what can be done with such power? We’ll find out later more about how these physical pages are managed when we deep-dive into our results.

I chose to go an easy route here by utilizing DLL injection and shared memory maps to pass information needed to process B from process A.

The first process (process A) allocates physical pages and a virtual window and writes data to that virtual window. Process A then creates a named mapped section to write the page frame number array we received from our call to AllocateUserPhysicalPages(). Process A spawns process B and injects a DLL that reads the mapped data, allocates physical pages and attempts to remap the physical pages to a new virtual window in process B.

Negative Results

So we already know the outcome of all this. It failed, specifically, we failed with the STATUS_CONFLICTING_ADDRESS error code. We expected it to fail because the documentation said it would. However, we aren't satisfied until we know WHY it failed. So let’s take a trip into the real reason we do research: the WHY.

Memory Allocation and Physical Pages

Earlier we mentioned the documentation stated the pages allocated through the AWE extensions are not included in the process’s working set. What is the working set?

Every process in the Windows OS upon execution is given a minimum and maximum working set; this set defines the amount of physical pages the process is allowed.

According to the Windows Internals Book:

“Every process starts with a default working set minimum of 50 pages and a working set maximum of 345 pages.”

These aren't hard limits however, and if needed the memory manager will allow an application to exceed its maximum limits if system resources permit. It will also trim them below the low limit if more resources are required by the system.

We mention this because the working set will typically contain pages that are committed through calls by the usermode program such VirtualAlloc(). AWE requires a call to VirtualAlloc, with the required MEM_PHYSICAL allocation type, and a subsequent call to NtAllocateUserPhysicalPages. The pages are not included in the working set, however they are still added to the Page Database and managed via some other page management system.

So if AWE pages are not in the working set, where exactly is it managed? We’ll need to go into the kernel and look at how the Virtual Memory is managed for AWE regions.

The Undocumented AWE Structure

The first difference with AWE is our call to VirtualAlloc and the MEM_PHYSICAL allocation type. This allocation creates a unique VAD entry in our process' VAD Tree. Below is the VAD tree for our RoundThree.exe. Take note of the two VAD entries marked as AWE regions.


When the call to NtAllocateUserPhysicalPages is made we can examine the disassembly and observe what happens. The function applies basic security and address boundaries checks (ensuring the SeLockMemoryPrivilege is enabled for the current process and the incoming virtual address is valid user-mode address).

NtAllocateUserPhysicalPages will reserve the number of physical pages our user-mode program requests. It does this by reserving physical pages through MiAllocatePagesForMdl(); the pages frames returned are filled into the user-mode buffer.

A structure named AweInfo is then allocated in the EPROCESS of the calling process. The AweInfo structure is undocumented and contains information about the physical pages that will be used in our AWE region. Among the data contained in it are the number of physical pages being requested, and a pointer to a (shadow) VAD Tree describing the virtual address windows we created using VirtualAlloc(). More specifically, it’s a pointer to an _MM_AVL_TABLE with MMADDRESS_NODEs describing the AWE virtual ranges. We can see from our WinDbg output below what the AweInfo structure looks like initially in memory. No AVL_TABLE has been created at this point, this occurs later during NtMapUserPhysicalPages.

0: kd> dx -r1 ((nt!_EPROCESS *)0x8581f508)->AweInfo
((nt!_EPROCESS *)0x8581f508)->AweInfo : 0x8581c7b0 [Type: void *]

0: kd> dd 0x8581c7b0
8581c7b0  8588a000 0000000c 00000000 8559d0f0
8581c7c0  00000000 00000000 00000000 00000000
8581c7d0  00000000 00000000 00000000 00000000
8581c7e0  00000000 00000000 00000000 00000000
8581c7f0  00000000 00000000 00000000 00000000
8581c800  00000000 00000000 00000000 00000000
8581c810  00000000 00000000 00000000 00000000
8581c820  00000000 00000000 00000000 00000000


When NtMapUserPhysicalPages is called, an _MM_AVL_TABLE is created and populated with the virtual addresses we’ll be mapping physical pages to.

NtMapUserPhysicalPages proceeds to map the physical pages (PFNs) to the virtual windows and assigns them a backing PTE (Page Table Entry). Below is a pointer to the AVL_TABLE populated when MapUserPhysicalPages is called (0x8558f328).

0: kd> dd 0x8581c7b0
8581c7b0  8588a000 0000000c 00000000 8559d0f0
8581c7c0  00000000 8558f328 00000000 00000000
8581c7d0  00000000 00000000 00000000 00000000
8581c7e0  00000000 00000000 00000000 00000000
8581c7f0  00000000 00000000 00000000 00000000
8581c800  00000000 00000000 00000000 00000000
8581c810  00000000 00000000 00000000 00000000
8581c820  00000000 00000000 00000000 00000000


We can decode the new pointer at offset 0x24 now as an MM_AVL_TABLE. Each entry is an MMADDRESS_NODE describing our AWE virtual addresses.
 

0: kd> dt nt!_MM_AVL_TABLE 8558f328
       +0x000 BalancedRoot     : _MMADDRESS_NODE
       +0x014 DepthOfTree      : 0y11000 (0x18)
       +0x014 Unused           : 0y101
       +0x014 NumberGenericTableElements : 0y100001010011110011111101 (0x853cfd)
       +0x018 NodeHint         : 0x00000003 Void
       +0x01c NodeFreeHint     : (null)

0: kd> dx -id 0,0,ffffffff85175ac8 -r1 (*((ntkrpamp!_MMADDRESS_NODE *)0xffffffff8558f328))
(*((ntkrpamp!_MMADDRESS_NODE *)0xffffffff8558f328))                 [Type: _MMADDRESS_NODE]
        [+0x000] u1               [Type: <unnamed-tag>]
        [+0x004] LeftChild        : 0x0 [Type: _MMADDRESS_NODE *]
        [+0x008] RightChild       : 0x851fd620 [Type: _MMADDRESS_NODE *]
        [+0x00c] StartingVpn      : 0xd0 [Type: unsigned long]
        [+0x010] EndingVpn        : 0xdc [Type: unsigned long]

0: kd> dx -r1 ((ntkrpamp!_MMADDRESS_NODE *)0x851fd620)
((ntkrpamp!_MMADDRESS_NODE *)0x851fd620)                 : 0x851fd620 [Type: _MMADDRESS_NODE *]
        [+0x000] u1               [Type: <unnamed-tag>]
        [+0x004] LeftChild        : 0x0 [Type: _MMADDRESS_NODE *]
        [+0x008] RightChild       : 0x0 [Type: _MMADDRESS_NODE *]
        [+0x00c] StartingVpn      : 0xe0 [Type: unsigned long]
        [+0x010] EndingVpn        : 0xec [Type: unsigned long]


It’s worth noting that each PFN entry related to our AWE region is marked by a field in the MMPFN structure. The AweReferenceCount is set to 1 if this page is related to an AWE allocation. Below is an entry in the PFN database that contains an AweReferenceCount of 1 (offset 0x10).
 

1: kd> dt nt!_MMPFN 84D51A04
       +0x000 u1               : <unnamed-tag>
       +0x004 u2               : <unnamed-tag>
       +0x008 PteAddress       : 0xc00016d8 _MMPTE
       +0x008 VolatilePteAddress : 0xc00016d8 Void
       +0x008 Lock             : 0n-1073735976
       +0x008 PteLong          : 0xc00016d8
       +0x00c u3               : <unnamed-tag>
       +0x010 OriginalPte      : _MMPTE
       +0x010 AweReferenceCount : 0n1
       +0x018 u4               : <unnamed-tag>


In the case of NtMapUserPhysicalPages with a NULL parameter for the PfnArrays, the function will traverse the PfnDatabase and clear the relevant pages backing PTE’s. The page frames can then be used to map a new AWE window.

Why We ‘Failed’

Now that we know a little more we can understand why our proof of concept (POC) failed. During the mapping of the physical pages, there are a number of sanity checks when looking up the page frame numbers and the virtual address being mapped. One such lookup is that the virtual address being mapped exists in our AweInfo structure; this check fails since we could not guarantee our VirtualAlloc call in Process B to match Process A’s. Furthermore, we would probably also fail a PTE lookup since we did not provide a valid page frame number array to NtMapUserPhysicalPages in process B.

So What Next?

While we didn't see success on our first try, it shouldn't discourage us from digging a bit deeper into abusing memory-related APIs. This is just one potential abuse that did not work out, but one thing I noticed was the low amount of lock primitives used when remapping was performed. Also, NtFreeUserPhysicalPages was never mentioned or looked into, could we abuse the timing of how and when the Windows Memory Manager decides to actually free up the physical pages (the pages still contain data we wrote when the previous window is unmapped)?

I hope this tale of failure and knowledge encourages other researchers to go out there and fail to learn (and share!). Eventually the knowledge gained will serve its purpose in future endeavors.

Happy Hunting.