Virtual Memory
Virtual Memory
This article covers the fundamentals of virtual memory and paging in Windows
Virtual Memory is a very important concept for understanding the bigger picture of OS internals, and crucial in relation to malware. It is a memory management mechanism with several benefits. It is used in all modern operating systems, and has been used for many decades.
Why it exists
Virtual memory addresses several important issues; primarily security, fragmentation, and resource efficiency.
The security angle comes from the fact that if all programs shared the same memory space, a program could accidentally overwrite memory used by another program, or malware could tamper with other programs easily. Programs could even overwrite the operating system's code. Virtual memory together with the user-mode/kernel-mode split takes care of this issue.
Fragmentation refers to memory with gaps. Not all memory will be contigous, especially as it comes and goes; it is going to leave gaps. Let's say you want to load 500MB of data into memory, and there is 600MB of free memory, but it is spread out in smaller chunks. You cannot fit the data contigously into any single block, so how do you use the memory? Virtual memory and paging solves this.
The other major angle is resource efficiency. There are several points to this. Virtual memory allows programs to use shared libraries, i.e. reused code which can point to the same underlying physical memory, saving space both on disk and in memory. You can also extend memory with a "swap file" on disk, sacrificing a bit of speed for more physical memory. However the largest impact comes from something called demand paging. It allows the system to load only the memory which is needed at that time. This is expanded upon in the paging section.
How it works
Virtual Memory
Virtual memory creates the illusion of a program having access to all the memory. It abstracts away raw memory management. Instead of a process directly accessing physical memory, the operating system provides a virtual memory space, which is then mapped to the physical memory. Since a process doesn't access physical memory directly, it does not need to worry about other program's memory.
Virtual memory is memory access indirection; essentially a translation scheme from a program's virtual memory to the real physical memory. Nothing is actually stored in virtual memory, everything is on physical memory or on disk
Paging
Virtual memory is split into small fixed size chunks called pages. Similarly, physical memory is split into small chunks called frames. A page may or may not reside in physical memory.
On Windows the default page size is 4kB, but larger pages can also be used.
Paging fixes the issue with fragmentation, as the pages can be spread out in any way. Ironically paging introduces a similar fragmentation issue given the page granularity: you must use full pages, and therefore the data in a page may not cover the entire page, leaving parts of memory unused.
Paging also allows for the operating system to only load the memory which is needed at that time. A page of virtual memory may be marked as used, but not present in physical memory. When you initially access that memory, it will create what is called a 'page fault', and the operating system will then load that data into physical memory. This is called demand paging.
When a program starts, only a small portion is actually put on the physical memory. As the program continues executing, it will load more memory to physical memory, as is needed. Similarly, when you manually allocate virtual memory, or map a file, it is put onto physical memory later on first access.
Notes
Memory pages have a type and protection. The protection tells what can be done with the memory (e.g. execute, read). The type indicates how it was allocated. Generally PE files loaded by the OS loader are marked MEM_IMAGE, file mappings (e.g. CreateFileMapping) are marked MEM_MAPPED, and manual allocations (e.g. VirtualAlloc) are marked as MEM_PRIVATE.
Type definitions from MSDN
| Action | Local Process | Remote Process |
|---|---|---|
| Allocate memory | VirtualAlloc, HeapAlloc ... | VirtualAllocEx, VirtualAlloc2, NtAllocateVirtualMemory, NtAllocateVirtualMemoryEx |
| Change memory protection | VirtualProtect | VirtualProtectEx, NtProtectVirtualMemory |
| Read process memory | no API call needed | ReadProcessMemory, NtReadVirtualMemory, NtReadVirtualMemoryEx |
| Write process memory | no API call needed | WriteProcessMemory, NtWriteVirtualMemory |
| Create file mapping (create object) | CreateFileMapping, CreateFileMappingNuma, NtCreateSection, NtCreateSectionEx | no such API; no need |
| Map section object to memory | MapViewOfFile, MapViewOfFileEx | MapViewOfFile2, MapViewOfFile3, MapViewOfFileNuma2, NtMapViewOfSection, NtMapViewOfSectionEx |
The best resource to read about the windows implementation of memory management in precise detail is the book "Windows Internals Part 1, 7th edition"