Date: March 6, 2025

Topic: Global Memory Systems

Recall

If a process page faults, the page it is looking for is not in pmem and usually we need to go to disk to get it. For GMS, when a fault occurs, we can additionally try to find the page in the cluster memory.

GMS only holds clean copies, and if copies are dirty, they are written onto disk. This prevents data loss in case of failures.

Notes

Context for Global Memory Systems

Global Memory System uses cluster memory for paging across the network and integrates it in memory hierarchy, so we may use peer memory for paging across LAN

Access to remote memory in caches may be faster than access to an actual electromechanical local disk.

• Virt. add. space running on a processor is much larger than phys. mem. allocated for it
• VMM gives illusion that all VA space is in Pmem (but only a portion of VA is in Pmem → working set of process)
  • Supports illusion for the process by paging in and out from the disk the pages accessed by a process at any point of time
• When a node is connected on LAN to other peer nodes on the same LAN, memory pressure differs across different nodes
  • If a node experiences memory pressure, can we use idle cluster memory for paging in and out the working set of processes on a node rather than go to disk
  • With high bandwidth links, sending or fetching a page from remote memory is faster than sending to local disk
• Hence, paging in and out to idle peer memories is a good solution when we have memory pressure
• For Virtual Memory Manager: VA → PA or disk
• For Global Memory System: VA → PA or cluster mem. or disk
• GMS doesn’t get in the way of writing to disk, disk always has copy of all pages
  • Only pages in the cluster memories are pages that have been paged out and not dirty
  • If there are dirty page copies, they are written onto the disk like a regular computer system
• GMS doesn’t add any new causes for worrying about failures
  • If a node fails, we only lose clean copies belonging to a particular process that may have been in the node’s memory
  • Those pages are also on the disk, so the disk always has copies of all pages
• VMM that is part of GMS can evict pages from pmem to make room
  • VMM goes out on the network to find idle peer memory and put the page in the peer memory
  • When page is later needed, VMM can quickly fetch it back from peer

In GMS, a single application can span across multiple nodes, where different nodes may need the same pages. GSM allows such sharing, and it will be in the local part of the pmem.

When the page is not used, it will be put in the global part, which acts like a disc and hence makes the page private.

The local and global portions change dynamically according to memory pressure on the particular node.

GMS Basics

• Cache refers to pmem (DRAM) not processor cache

• Sense of community to handle page faults at a node

  • If every node has own process, then everything is private, no community needed
  • However, possible for an application to span multiple nodes so pages can be shared

• Peer memories are used as a supplement to the disk, where the physical memory of a node looks like:

  

  • Local contains current working set of currently executing processes, things node needs to keep the current processes happy
  • Global is similar to a disk and where community service comes in. This portion can be used for holding pages from fellow nodes
  • Split of Local and Global memory is dynamic in response to current memory pressure

• Pages can either be private or shared

  • Shared: When a page is shared, it can be in Local part of multiple peers since multiple peers are actively using a page
    • Pages in local memory can be private or shared, depending if the particular page is being actively shared
  • Private: If page is in Global part, it is guaranteed to be private as Global part is similar to a disk
    • When something is swapped out (not in active use), it is thrown onto the disk
    • Similarly when swapping out in GMS, it is thrown into the Global cache, making it private

• If multiple processes share a page, VMM has no concern about coherence of the pages being shared

  • Coherence for shared pages is outside GSM and is an application problem
  • GMS only cares about remote paging

• GMS picks the globally oldest page for replacement (Least Recently Used)

  • GMS has to manage age information

<aside> 📌 SUMMARY: A Global Memory System (GMS) distributes memory pressure across multiple nodes in a distributed system by using strategies such as geriatric management, which ensures older, less-frequently used pages are migrated to idle nodes or disk storage. Each node maintains a fixed sum of local and global cache sizes—where the local cache stores the node’s working set and the global cache provides community memory resources for other nodes experiencing pressure. Coordination epochs are initiated by the node predicted to be the most idle to minimize overhead. GMS implementation integrates closely with the Virtual Memory Manager (VMM), Unified Buffer Cache (UBC), Pageout Daemon, and Free List. When handling a page fault, three data structures are traversed in the sequence: Virtual Address (VA) → User/Process ID (UID) → Page Ownership Directory (POD) → Global Cache Directory (GCD) → Page Frame Directory (PFD). The GCD maintains a dynamic global mapping of pages, preventing static POD-to-PFD mapping issues, especially important during periods of high page traffic.

</aside>