KR102444562B1 - hierarchical memory system - Google Patents

Abstract

Apparatus, systems, and methods corresponding to hierarchical memory systems are described. The logic circuitry may reside on the permanent memory device, thereby reducing latency associated with transferring data between the logic circuitry and the permanent memory device. Logic circuitry on the persistent memory device may include an address register configured to store logical addresses corresponding to the stored data. The logic circuitry receives the redirected request (eg, directed to the non-persistent memory device, prior to the redirect) to retrieve the portion of data stored in the persistent memory device, and upon receipt of the request to retrieve the portion of data. In response, determine a physical address corresponding to the portion of data based on the logical address stored in the address register, and cause the data to be retrieved from the persistent memory device.

Description

hierarchical memory system

BACKGROUND This disclosure relates generally to semiconductor memory and methods, and more specifically to apparatus, systems, and methods relating to hierarchical memory systems.

Memory devices are typically provided as internal semiconductor, integrated circuitry within computers or other electronic systems. Many different types of memories include volatile and non-volatile memory. Volatile memory may require power to hold its data (eg, host data, error data, etc.) and, in particular, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and synchronous dynamic random access memory (SDRAM). Non-volatile memories can provide persistent data by retaining stored data when power is not applied, particularly NAND flash memory, NOR flash memory, variable memory, such as phase change random access memory (PCRAM), resistive random access memory (RRAM), and magnetic random access memory (MRAM), such as rotational torque transfer random access memory (STT RAM).

Memory devices may be coupled to a host (eg, a host computing device) to store data, commands, and/or instructions for use by the host during operation of the computer or electronic system. For example, data, commands, and/or instructions may be transferred between the host and the memory device(s) during operation of a computing or other electronic system.

1 is a functional block diagram in the form of a device including logic circuitry in accordance with multiple embodiments of the present disclosure;
2 is a functional block diagram in the form of a computing system including logic circuitry residing on a persistent memory device in accordance with multiple embodiments of the present disclosure.
3 is a functional block diagram in the form of a computing system including logic circuitry residing on a persistent memory device in accordance with several embodiments of the present disclosure.
4 is a flowchart illustrating a data read operation in accordance with various embodiments of the present disclosure.
5 is a flowchart illustrating predictive data write operations in accordance with multiple embodiments of the present disclosure.
6 is a flow diagram illustrating an exemplary method for logic circuitry in a memory in accordance with multiple embodiments of the present disclosure.

Apparatus, systems, and methods corresponding to hierarchical memory systems are described. The logic circuitry may reside on the permanent memory device, thereby reducing latency associated with transferring data between the logic circuitry and the permanent memory device. An exemplary apparatus includes a permanent memory device and logic circuitry residing on the permanent memory device. The logic circuitry includes an address register configured to store logical addresses corresponding to data stored in the persistent memory device. The logic circuitry retrieves the portion of data stored in the persistent memory device to receive a redirected request (eg, directed to the non-persistent memory device, prior to the redirect) to retrieve the portion of data stored in the persistent memory device. in response to receiving the request to determine a physical address corresponding to the portion of data based on the logical address stored in the address register; and based on the determined address, cause data to be retrieved from the persistent memory device.

Computing systems utilize various types of memory resources during operation. For example, the computing system may utilize a combination of volatile (eg, random access memory) memory resources and non-volatile (eg, storage) memory resources during operation. In general, volatile memory resources may operate at a much higher speed than non-volatile memory resources, and may have a longer lifetime than non-volatile memory resources; Volatile memory resources are typically more expensive than non-volatile memory resources. As used herein, a volatile memory resource may alternatively be referred to as a “non-persistent memory device,” while a non-volatile memory resource may alternatively be referred to as a “persistent memory device.”

However, a persistent memory device may more broadly refer to the ability to access data in a persistent manner. As an example, in the context of persistent memory, the memory device provides a plurality of logical-to-physical mapping or translation data and/or lookups into a memory array to track the location of data in the memory device, independent of whether the memory is non-volatile or not. You can save tables. Furthermore, persistent memory devices include the ability to service commands for successive processes (eg, by using logical-to-physical mappings, lookup tables, etc.), so that in addition to using non-volatile It can refer to both volatile.

These characteristics may require a trade-off in computing systems to provision the computing system with appropriate resources to function in accordance with the ever-increasing demands of consumers and computing resource providers. For example, in a multi-user computing network (eg, a cloud-based computing system deployment, a software-defined data center, etc.), a relatively large amount of volatile memory may be provided to provisioned virtual machines running in the multi-user network. . However, as is common in some approaches, by relying on volatile memory to provide memory resources to a multi-user network, especially as users of the network require increasingly large pools of computing resources to be made available, memory resource The cost associated with provisioning them to the network may increase.

Furthermore, in approaches that rely on volatile memory to provide memory resources for provisioning virtual machines in a multi-user network, when volatile memory resources are exhausted (eg, volatile memory resources are allocated to users of the multi-user network) ), additional users may not be added to the multi-user network until additional volatile memory resources are available or added. This may cause potential users to turn away, which may result in a loss of revenue that could be generated if additional memory resources were available for a multi-user network.

Volatile memory resources, such as dynamic random access memory (DRAM), tend to operate in a deterministic manner, while storage class memories (eg, NAND flash memory devices, solid state drives, resistive variable memory devices, etc.) ), non-volatile memory resources tend to behave in a non-deterministic manner. For example, due to error correction operations, encryption operations, RAID operations, etc. performed on data retrieved from storage class memory devices, between requesting data from a storage class memory device and data being available. The amount of time of may vary from read to read, thereby making data retrieval from storage class memory devices non-deterministic. In contrast, the amount of time between requesting data from a DRAM device and when data is available can be held fixed per read, thereby making data retrieval from the DRAM device deterministic.

Also, because of the distinction between the deterministic behavior of volatile memory resources and the non-deterministic behavior of non-volatile memory resources, data transferred to and from the memory resource is generally transferred to a specific interface (e.g., bus) tour. For example, data transferred to and from a DRAM device is typically carried over a double data rate (DDR) bus, while data transferred to and from a NAND device is typically carried over a peripheral component interconnect express (PCIe) bus. transmitted through However, as will be appreciated, examples of interfaces through which data may be transferred to and from volatile and non-volatile memory resources are not limited to these specific enumerated examples.

Because of the different behaviors of non-volatile memory devices and volatile memory devices, some approaches choose to store certain types of data in either volatile or non-volatile memory. This may mitigate issues that may arise, for example, due to the deterministic behavior of volatile memory devices compared to the non-deterministic behavior of non-volatile memory devices. For example, computing systems in some approaches store small amounts of regularly accessed data in volatile memory devices during operation of the computing system, while larger or less frequently accessed data stores in non-volatile memory devices. do. However, in a multi-user network deployment, the majority of data may be stored in volatile memory devices. In contrast, embodiments herein may allow for data storage and retrieval from non-volatile memory devices deployed in a multi-user network.

As described herein, some embodiments of the present disclosure provide for computing that is non-volatile, and thereby data from a non-deterministic memory resource is transferred over an interface that is limited to use by the volatile and deterministic memory resource in other approaches. It's about systems. For example, in some embodiments, the data is in some approaches a non-volatile, such as a NAND flash device, via an interface, such as a DDR interface, reserved for data transfer to and from a volatile deterministic memory resource. Non-deterministic memory resources, resistive variable memory devices such as phase change memory devices and/or resistive memory devices (eg, three-dimensional crosspoint (3D XP) memory devices), solid state drives (SSDs), self-selecting memories (self -selecting memory, SSM) devices, and the like. Thus, in contrast to approaches in which volatile, deterministic memory devices are used to provide main memory to a computing system, embodiments herein allow non-volatile, non-deterministic memory devices to provide at least a portion of main memory for a computing system. It may be allowed to be used as

In some embodiments, data may be transferred from a non-volatile memory resource to an interim cache (eg, a small static random access memory (SRAM) cache) or buffer, and then made available to the application that requested the data. have. As described herein, by storing data that is typically provided in a deterministic manner in a non-deterministic memory resource and allowing access to that data, it is substantially more practical than approaches that operate using, for example, volatile memory resources. Computing system performance can be improved by allowing a greater amount of memory resources to be made available to a multi-user network at a reduced cost.

Furthermore, multiple embodiments of the present disclosure may reduce multiple steps typically performed when transferring data to/from non-volatile memory resources by tightly coupling multiple components of a hierarchical memory system together. For example, in many embodiments, the logic circuitry that coordinates routing data requests between the host and non-volatile memory resources (and temporarily stores data transferred to/from non-volatile memory resources) may include permanent memory It may be disposed within a memory device having non-volatile memory resources, such as the device. Accordingly, in many embodiments of the present disclosure, data having a relatively larger size is transferred to/from the persistent memory device via a reduced number of external buses (eg, data buses external to the persistent memory device). can be sent, which improves the overall processing speed of read/write requests associated with the persistent memory device.

To enable embodiments of the present disclosure, visibility into non-volatile memory resources may be obfuscated for various devices of the computing system in which the hierarchical memory system is deployed. For example, a host(s), network interface card(s), virtual machine(s), etc., deployed in a computing system or multi-user network may determine whether data is stored by volatile or non-volatile memory resources of the computing system. It may not be possible to distinguish whether it is stored by For example, host(s), network interface card(s), virtual machine(s), etc. may access data in a way that cannot distinguish whether the data is stored by a volatile or non-volatile memory resource. Hardware circuitry capable of registering corresponding addresses may be disposed in the computing system.

As described in greater detail herein, a hierarchical memory system intercepts redirected data requests, registers an address associated with the requested data in logic circuitry (where the hardware circuitry is not backed up by its own memory resources to store data). notwithstanding), hardware circuitry (eg, logic circuitry) capable of using the logic circuitry to map addresses registered in the logic circuitry to physical addresses corresponding to data in the non-volatile memory device. have.

In the following detailed description of the disclosure, reference is made to the accompanying drawings, which form a part hereof, and which show by way of example how one or more embodiments of the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments of the present disclosure, and as other embodiments may be utilized and process, electrical, and structural changes may be made without departing from the scope of the disclosure. should be understood as possible.

As used herein, designators such as “N”, “M”, etc. for reference signs, particularly in the drawings, indicate that a number of specific features so designated may be included. Also, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. As used herein, singular expressions are used unless the context clearly dictates otherwise. It can include both singular and plural objects. Also, “plurality”, “at least one”, and “one or more” (eg, multiple memory banks) may refer to one or more memory banks, while “plurality” means that these are more than one. It is meant to refer to many things.

Furthermore, the words "may" and "may" are used throughout this application in a permissive sense (i.e., likely to, capable of), rather than a necessary (i.e., should) sense. is used as The term "comprises", and its derivatives, means "including, but not limited to". The terms "coupled" and "coupling" are physically connected directly or indirectly or for access to and movement (transmission) of commands and/or data, as the context appropriate. means The terms “data” and “data values” may be used interchangeably herein and have the same meaning, as appropriate in the context.

The drawings herein follow a numbering convention in which the first number or numbers correspond to the reference number and the remaining numbers identify an element or component in the drawing. Similar elements or components between different figures may be identified by use of like numbers. For example, 104 may refer to element “04” in FIG. 1 , and similar elements may be referred to as 204 in FIG. 2 . A plurality of similar elements or components or groups of similar elements or components may be collectively referred to herein by a single element number. For example, the plurality of reference elements 106-1, 106-2, ..., 106-N (eg, 106-1 to 106-N) may be collectively referred to as 106 . As will be appreciated, elements presented in the various embodiments herein may be added, exchanged, and/or removed to provide numerous additional embodiments of the present disclosure. Also, the proportions and relative scales of elements provided in the drawings are intended to illustrate specific embodiments of the present disclosure and should not be taken in a limiting sense.

1 is a functional block diagram in the form of a computing system 100 including an apparatus including logic circuitry 104 in accordance with multiple embodiments of the present disclosure. As used herein, “apparatus” means any of various structures or combinations of structures, such as, for example, a circuit or circuitry, die or dies, module or modules, device or devices, or system or systems. may refer to, but is not limited to. In some embodiments, logic circuitry 104 may be provided as a field programmable gate array (FPGA), application specific integrated circuit (ASIC), multiple discrete circuit components, etc., alternatively referred to herein as "logic circuitry" may be referred to.

Logic circuitry 104 may include a memory resource 102 , as shown in FIG. 1 , which may include a read buffer 103 , a write buffer 105 , and/or input/output I/O device access components. (107) may be included. In some embodiments, the memory resource 102 may be a random access memory resource, such as a block RAM, which may enable data to be stored within the logic circuitry 104 in embodiments where the logic circuitry 104 is an FPGA. . However, embodiments are not so limited, and the memory resource 102 may include various registers, caches, memory arrays, latches, and SRAM, DRAM, EPROM, or data stored external to the logic circuitry 104 . other suitable memory technologies capable of storing data, such as bit strings containing registered addresses corresponding to physical locations. Memory resource 102 is internal to logic circuitry 104 and is generally external to logic circuitry 104 , such as permanent and/or non-persistent memory resources that may be external to logic circuitry 104 . smaller than

The read buffer 103 may include a portion of the memory resource 102 reserved for storing data received by the logic circuitry 104 but not processed by the logic circuitry 104 . In some embodiments, the read buffer 103 may be about 4 kilobytes (KB) in size, although embodiments are not limited to this particular size. The read buffer 103 may buffer data to be registered in one of the address registers 106 - 1 to 106-N.

The write buffer 105 may include a portion of the memory resource 102 reserved for storing data awaiting transmission to a location external to the logic circuitry 104 . In some embodiments, the write buffer 105 may be about 4 kilobytes (KB) in size, although embodiments are not limited to this particular size. The write buffer 103 may buffer data registered in one of the address registers 106-1 to 106-N.

I/O access component 107 is used herein to store data corresponding to access to a component external to logic circuitry 104 , such as I/O device 210/310 shown in FIGS. 2 and 3 . It may include a portion of the reserved memory resource 102 . The I/O access component 107 may store data corresponding to addresses of the I/O device, which may be used to read and/or write data to and from the I/O device. Also, in some embodiments, I/O access component 107 is a hypervisor (eg, hypervisor 312 shown in FIG. 3 ), as described in greater detail herein with respect to FIG. 3 . may receive, store, and/or transmit data corresponding to the state of

The logic circuitry 104 may further include a memory access multiplexer (MUX) 109 , a state machine 111 , and/or a hierarchical memory controller 113 (or, for brevity, a “controller”). As shown in FIG. 1 , the hierarchical memory controller 113 may include a plurality of address registers 106 - 3 through 106-N and/or an interrupt component 115 . Memory access MUX 109 may include one or more logic gates and may include circuitry that may be configured to control data and/or address busing to logic circuitry 104 . For example, the memory access MUX 109 may send messages to and from the memory resource 102 , as well as a hierarchical memory controller 113 and/or a state machine ( 111) can be communicated with.

In some embodiments, MUX 109 may redirect incoming messages and/or commands from a received host (eg, host computing device, virtual machine, etc.) to logic circuitry 104 . For example, MUX 109 may respond to an access request (eg, received over an interface such as interface 208/308 shown in FIGS. 2 and 3 herein) from an I/O device). One of the address registers for the read buffer 103 and/or the write buffer 105 (eg, the address register, which may be the BAR4 area of the hierarchical memory controller 113 as described below) (106-N)).

MUX 109 may also redirect requests (eg, read requests, write requests) received by logic circuitry 104 . In some embodiments, requests are made to a hypervisor communicatively coupled to logic circuitry 104 (eg, hypervisor 312 shown herein in FIG. 3 ), a bare metal server, or received by the logic circuitry 104 from the host computing device. These requests are sent by the MUX 109 from the read buffer 103 , the write buffer 105 , and/or the I/O access component 107 to an address register (eg, a hierarchical memory controller as described below). address register 106-2, which may be the BAR2 region of 113).

MUX 109 may redirect these requests as part of an operation to determine an address in address register(s) 106 to be accessed. In some embodiments, MUX 109 is responsive to an assertion of a hypervisor interrupt (eg, an interrupt asserted to a hypervisor coupled to logic circuitry 104 generated by interrupt component 115 ). These requests may be redirected as part of the operation to determine the address in the address register(s) to be accessed.

A request is made to data stored external to logic circuitry 104 (eg, to a location external to logic circuitry 104 (eg, intermediate memory components 220/320 shown in FIGS. 2 and 3 herein). and/or data associated with the address written to the array 222/322 of the persistent memory device 216/316), the MUX 109 retrieving the data, a write buffer of the data. transfer to 105, and/or a location external to logic circuitry 104 of data (eg, a memory cell of a persistent memory device, such as persistent memory device 216/316 shown in FIGS. 2 and 3 herein) of arrays). In response to determining that the request corresponds to data being read from a location external to logic circuitry 104 (eg, from a persistent memory device), MUX 109 retrievals data, read buffer 103 of data. transfer to and/or transfer of data or address information associated with the data to a location internal to the logic circuitry 104 , such as the address register(s) 106 .

As a non-limiting example, when the system logic 104 receives a read request from an I/O device, the MUX 109 selects the appropriate messages to transmit from the logic 104 to persistent memory via the hypervisor. It may enable retrieval of data from the device. For example, the MUX 109 may enable generation of an interrupt using the interrupt component 115 , cause the interrupt to be asserted on the hypervisor, and/or read data received from the persistent memory device into the read buffer 103 . ) and/or respond to the I/O device with an indication that the read request has been fulfilled. In a non-limiting example where logic circuitry 104 receives a write request from an I/O device, MUX 109 selects appropriate messages to transmit from logic circuitry 104 to the persistent memory device via the hypervisor. data can be transmitted. For example, the MUX 109 may enable generation of an interrupt using the interrupt component 115 , cause the interrupt to be asserted on the hypervisor, and/or write data to be transferred to the persistent memory device into the write buffer 105 . ) and/or respond to the I/O device with an indication that the write request has been fulfilled.

State machine 111 may include one or more processing devices, circuit components, and/or logic configured to perform operations on inputs and generate outputs. In some embodiments, state machine 111 may be a finite state machine (FSM) or hardware state machine that may be configured to receive variable inputs and generate a resulting output based on the received inputs. For example, state machine 111 may send access information (eg, "I/O ACCESS INFO") to and from memory access multiplexer 109, as well as interrupt configuration information (eg, “INTERRUPT CONFIG”) and/or interrupt request messages (eg, “INTERRUPT REQUEST”) may be sent to and from the hierarchical memory controller 113 . In some embodiments, state machine 111 may also send control messages (eg, “MUX CTRL”) to and from memory access multiplexer 109 .

The ACCESS INFO message may include information corresponding to a data access request received from an I/O device external to the logic circuitry 104 . In some embodiments, the ACCESS INFO may include logical addressing information corresponding to data to be stored in the persistent memory device or addressing information corresponding to data to be retrieved from the persistent memory device.

The INTERRUPT CONFIG message may be asserted by the state machine 111 on the hierarchical memory controller 113 to construct appropriate interrupt messages to be asserted outside the logic circuitry 104 . For example, when logic circuitry 104 asserts an interrupt on a hypervisor coupled to logic circuitry 104 as part of fulfilling a redirected read or write request, the INTERRUPT CONFIG message indicates that the operation is a persistent memory device. may be generated by the state machine 111 to generate an appropriate interrupt message based on whether the operation is to retrieve data from or write the data to a persistent memory device.

The INTERRUPT REQUEST message may be generated by the state machine 111 and asserted on the interrupt component 115 to cause the interrupt message to be asserted on the hypervisor (or bare metal server or other computing device). As described in greater detail herein, interrupt 115 is an interrupt on the hypervisor to cause the hypervisor to prioritize data retrieval or writing of data to a persistent memory device as part of the operation of the hierarchical memory system. can be written

The MUX CTRL message(s) may be generated by the state machine 111 and asserted on the MUX 109 to control the operation of the MUX 109 . In some embodiments, the MUX CTRL message(s) may be asserted on the MUX 109 by the state machine 111 as part of performance of the MUX 109 operations described above (or vice versa). ).

The hierarchical memory controller 113 may include a core, such as an integrated circuit, chip, system on chip, or combinations thereof. In some embodiments, the hierarchical memory controller 113 may be a peripheral component interconnect express (PCIe) core. As used herein, “core” refers to a reusable unit of logic, processor, and/or coprocessors that receives instructions and performs tasks or operations based on the received instructions.

The hierarchical memory controller 113 may include address registers 106 - 1 through 106-N and/or an interrupt component 115 . Address registers 106 - 1 through 106-N are memory used by logic circuitry 104 or computing system (eg, computing system 201/301 shown in FIGS. 2 and 3 herein). They may be base address registers (BARs) that can store addresses. At least one of the address registers (eg, address register 106 - 1 ) provides access to internal registers of logic circuitry 104 from an external location, such as hypervisor 312 shown in FIG. 3 . Memory addresses can be stored.

A different address register (eg, address register 106 - 2 ) may be used to store addresses corresponding to interrupt control, as described in more detail herein. In some embodiments, address register 106-2 may map direct memory access (DMA) read and DMA write control and/or status registers. For example, address register 106-2 includes the generation of one or more interrupt messages that may be asserted to the hypervisor as part of the operation of a hierarchical memory system, as described herein with respect to FIG. 3 . It may include addresses corresponding to descriptors and/or control bits for DMA command chaining that may be enabled.

Another of the address registers (eg, address register 106 - 3 ) corresponds to access to and from the hypervisor (eg, hypervisor 312 shown in FIG. 3 , herein). addresses can be stored. In some embodiments, access to and/or from the hypervisor may be provided via an Advanced eXtensible Interface (AXI) DMA associated with the logic circuitry 104 . In some embodiments, the address register may map addresses corresponding to data transferred via DMA (eg, AXI DMA) of logic circuitry 104 to locations external to logic circuitry 104 .

In some embodiments, at least one address register (eg, address register 106-N) is an I/O device to logic circuitry 104 (eg, I/O device 210 shown in FIG. 2 ). ) addresses corresponding to accesses. Address register 106-N may store addresses that are bypassed by DMA components associated with logic circuitry 104 . The address register 106-N may be provided so that addresses mapped thereto are not "backed up" by the physical memory location of the logic circuitry 104 . That is, in some embodiments, logic circuitry 104 corresponds to data stored in a persistent memory device (eg, persistent memory device 216 shown in FIG. 2 ) and data stored by logic circuitry 104 includes: It may consist of an address space that stores non-corresponding addresses. For example, the address register 106-N may be configured as a virtual address space that may store logical addresses corresponding to physical memory locations (eg, in a memory device) where data is stored.

In some embodiments, address register 106-N is an amount of address spaces corresponding to the size of a memory device (eg, persistent memory device 216/316 shown in FIGS. 2 and 3 herein). may include. For example, if the memory device includes one terabyte of storage, the address register 106-N may be configured to have an address space that may include one terabyte of address space. However, as noted above, the address register 106-N is configured to appear as if it does not actually contain 1 terabyte of storage, but instead has 1 terabyte of storage space.

Although not explicitly shown in FIG. 1 , the logic circuitry 104 may be coupled to a host computing system. The host computing system may include a system motherboard and/or backplane, and may include multiple processing resources (eg, one or more processors, microprocessors, or some other type of control circuitry). Host and device 100 may be, for example, server systems and/or high-performance computing (HPC) systems and/or parts thereof. In some embodiments, the computing system may have a von Neumann architecture, although embodiments of the present disclosure will not typically include one or more components (eg, CPU, ALU, etc.) associated with von Neumann architecture. It can be implemented in non-von Neumann architectures that can.

2 and 3 , logic circuitry 104 is hierarchical, such as a persistent memory device (eg, persistent memory devices 216/316 shown in FIGS. 2 and 3 herein). It may reside on a particular memory device in a memory system. As used herein, the term “resident on” refers to being physically located on a particular component. For example, logic circuitry 104 “resident on” permanent memory devices 216/316 refers to a state in which logic circuitry 104 is physically coupled to persistent memory devices 216/316. The term “resident on” may be used interchangeably herein with other terms such as “disposed on” or “located on”. As described herein, many embodiments of the present disclosure can reduce the latency associated with transferring data to/from the persistent memory device by placing logic circuitry within the persistent memory device. Since data transferred to/from a persistent memory device typically has a relatively large size, it is a hierarchical memory system (eg, hierarchical memory systems 201/301 shown in FIGS. 2 and 3 herein). ) can significantly improve the overall processing speed. Additional details of the logic circuitry residing on the persistent memory device are described with respect to FIGS. 2 and 3 .

2 is a functional block diagram in the form of a computing system 201 that includes logic circuitry 204 (eg, logic circuitry) residing on a persistent memory device 216 in accordance with various embodiments of the present disclosure. As shown in FIG. 2 , computing system 201 may include a persistent memory device 216 in which logic circuitry 204 resides. The logic circuitry 204 may be similar to the logic circuitry 104 shown in FIG. 1 . The computing system 201 may also include an input/output (I/O) device 210 , a non-persistent memory device 230 , an intermediate memory component 220 , and a memory management component 214 . between the I/O device 210 and the persistent memory device 216 (as well as the logic circuitry 204 ), the non-persistent memory device 230 , the intermediate memory component 220 , and/or the memory management component 214 . Communication may be enabled via interface 208 .

I/O device 210 may be a device configured to provide direct memory access via a physical address and/or a virtual machine physical address. In some embodiments, I/O device 210 may be a network interface card (NIC) or network interface controller, storage device, graphics rendering device, or other I/O device. I/O device 210 may be a physical I/O device, or I/O device 210 may be a virtualized I/O device 210 . For example, in some embodiments, I/O device 210 may be a physical card that is physically coupled to the computing system via an interface or bus, such as a PCIe interface or other suitable interface. In embodiments where the I/O device 210 is a virtualized I/O device 210 , the virtualized I/O device 210 may provide I/O functionality in a distributed manner.

Persistent memory device 216 may include multiple arrays of memory cells, such as array 222 . The arrays may be, for example, flash arrays with a NAND architecture. However, embodiments are not limited to a particular type of memory array or array architecture. Memory cells may be grouped into multiple blocks comprising multiple physical pages, for example. Multiple blocks may be included in a plane of memory cells, and an array may include multiple planes.

Persistent memory device 216 may include volatile memory and/or non-volatile memory. In many embodiments, the persistent memory device 216 may comprise a multi-chip device. A multi-chip device may include a number of different memory types and/or memory modules. For example, a memory system may include non-volatile or volatile memory on any type of module. In embodiments where persistent memory device 216 includes non-volatile memory, persistent memory device 216 may be a flash memory device such as NAND or NOR flash memory devices.

However, embodiments are not limited thereto, and the persistent memory device 216 may include other non-volatile memory devices, such as non-volatile random access memory devices (eg, NVRAM, ReRAM, FeRAM, MRAM, PCM), “recently "made" memory devices, such as resistive variable memory devices (eg, resistive and/or phase change memory devices, such as a 3D crosspoint (3D XP) memory device), self-selecting memory (SSM) memory devices comprising an array of cells, or the like, or combinations thereof. A resistive and/or phase change array of non-volatile memory, in conjunction with a stackable cross grid data access array, may perform bit storage based on changes in bulk resistance. Additionally, resistive and/or phase change memory devices, unlike many flash-based memories, cannot perform a write in-place operation in which a non-volatile memory cell can be programmed without the non-volatile memory cell being previously erased. can Self-selecting memory cells, in contrast to flash-based memories, may include memory cells having a single chalcogenide material that serves as both a storage element and a switch for the memory cell.

The persistent memory device 216 can provide a storage volume for the computing system 201 (eg, via the array 222 ), and thus the computing system 201 , the main for the computing system 201 . may be used as additional memory or storage throughout the memory, or combinations thereof. Although embodiments are not limited to a particular type of memory device, persistent memory device 216 may include RAM, ROM, SRAM DRAM, SDRAM, PCRAM, RRAM, and flash memory, among others. Furthermore, although a single persistent memory device 216 is shown in FIG. 2 , embodiments are not limited thereto, and the computing system 201 may include one or more persistent memory devices 216 , each of which is associated therewith. They may or may not have the same architecture. As a non-limiting example, in some embodiments, the persistent memory device 216 may include two separate memory devices of different architectures, such as a NAND memory device and a resistive variable memory device.

As described herein and shown in FIG. 2 , logic circuitry 204 may reside on a persistent memory device 216 . Accordingly, data transferred to (eg, to be stored in) the array 222 is (rather than using a data bus external to the persistent memory device 216 ) a data bus 218 internal to the persistent memory device 216 . ) from the logic circuit unit 204 to the array 222 . Similarly, data transferred from (eg, to be read from) array 222 may be transferred directly from array 222 to logic circuitry 204 via data bus 218 . This may improve the processing speed of read/write requests associated with data being transferred to/from the persistent memory device 216 , which may also improve the overall speed of the hierarchical memory system 201 .

Non-persistent memory device 230 may include volatile memory, such as an array of volatile memory cells. In various embodiments, non-persistent memory device 230 may comprise a multi-chip device. A multi-chip device may include a number of different memory types and/or memory modules. In some embodiments, non-persistent memory device 230 may serve as main memory for computing system 201 . For example, the non-persistent memory device 230 may be a dynamic random access (DRAM) memory device used to provide main memory to the computing system 230 . However, embodiments are not limited to non-persistent memory device 230 including a DRAM memory device, and in some embodiments, non-persistent memory device 230 may, in particular, include RAM, SRAM DRAM, SDRAM, PCRAM, and/or or other non-persistent memory devices such as RRAM.

Non-persistent memory device 230 may store data that may be requested by a host computing device as part of operation of computing system 201 , for example. For example, when the computing system 201 is part of a multi-user network, the non-persistent memory device 230 may be connected to host computing devices (eg, a virtual machine deployed in a multi-user network) during operation of the computing system 201 . ) can store data that can be transferred between

In some approaches, non-persistent memory, such as non-persistent memory device 230 , may store all user data accessed by a host (eg, a virtual machine deployed in a multi-user network). For example, due to the speed of non-persistent memory, some approaches rely on non-persistent memory to provision memory resources for virtual machines deployed in a multi-user network. However, in these approaches, costs can be an issue because non-persistent memory is generally more expensive than permanent memory (eg, persistent memory device 216 ).

In contrast, as described in greater detail below, embodiments herein may enable at least some data stored in non-persistent memory device 230 to be stored in persistent memory device 216 . This may enable additional memory resources to be provided to the computing system 201 , such as a multi-user network, at a lower cost than approaches that rely on non-persistent memory for user data storage.

The computing system 201 may include a memory management component 214 that may be communicatively coupled to a non-persistent memory device 230 and/or an interface 208 . In some embodiments, memory management component 214 is an input/output memory management unit (IO MMU) capable of communicatively coupling a direct memory access bus, such as interface 208 , to non-persistent memory device 230 . can However, embodiments are not limited thereto, and the memory management component 214 may be other types of memory management hardware that enables communication between the interface 208 and the non-persistent memory device 230 .

Memory management component 214 may map device visible virtual addresses to physical addresses. For example, memory management component 214 can map virtual addresses associated with I/O device 210 to physical addresses in non-persistent memory device 230 and/or persistent memory devices 216 . . In some embodiments, mapping of virtual entries associated with I/O device 210 may be facilitated herein by the read buffer, write buffer, and/or I/O access buffer shown in FIG. 1 .

In some embodiments, memory management component 214 reads a virtual address associated with I/O device 210 and/or sets the virtual address to a physical address in non-persistent memory device 230 or logical circuitry 204 . It can be mapped to an address in . In embodiments where the memory management component 214 maps the virtual I/O device 210 address to an address in the logic circuitry 204 , the memory management component 214 may The read request (or write request) may be redirected to the logic circuitry 204 , which transfers the virtual address information associated with the I/O device 210 read or write request to an address register (eg, address register 206-N). In some embodiments, address register 206-N may be a specific base address register of logic circuitry 204, such as a BAR4 address register.

The redirected read (or write) request may be sent from the memory management component 214 to the logic circuitry 204 via the interface 208 . In some embodiments, interface 208 may be a PCIe interface, and thus may pass information between memory management component 214 and logic circuitry 204 according to PCIe protocols. However, embodiments are not so limited, and in some embodiments, interface 208 may be an interface or bus that functions according to another suitable protocol.

After the virtual NIC address is stored in the logical circuitry 204 , data corresponding to the virtual NIC address may be written to the persistent memory device 216 . For example, data corresponding to a virtual NIC address stored in the logical circuitry 204 may be stored in a physical address location of the persistent memory device 216 . In some embodiments, transferring data to and/or from persistent memory device 216 may be enabled by a hypervisor, as described herein with respect to FIGS. 3-5 .

When data is requested by a host computing device, such as, for example, a virtual machine disposed in computing system 201 , the request transfers interface 208 from I/O device 210 by memory management component 214 . through the logic circuit unit 204 . Since the virtual NIC address corresponding to the physical location of the data in the persistent memory device 216 is stored in the address register 206-N of the logic circuitry 204, the logic circuitry 204 is used herein, in FIGS. 5 , in conjunction with the hypervisor, may enable retrieval of data from the persistent memory device 216 .

In some embodiments, when data that was stored in persistent memory device 216 is transferred from persistent memory device 216 (eg, data that was stored in persistent memory device 216 by a host computing device will be requested). ), the data may be transferred to the intermediate memory component 220 and/or the non-persistent memory device 230 before being provided to the host computing device. For example, since data sent to the host computing device may be sent in a deterministic manner (eg, via a DDR interface), the data may be transferred to the intermediate memory component 220 and/or before the data request is fulfilled. It may be temporarily transferred to a memory operating using a DDR bus, such as non-persistent memory device 230 .

3 is a functional block diagram in the form of a computing system including logic circuitry 304 residing on a persistent memory device 316 in accordance with various embodiments of the present disclosure. Logic circuitry 304 may be similar to logic circuitry 104/204 shown in FIGS. 1 and 2 . The computing system 301 may also include an I/O device 310 , a non-persistent memory device 330 , an intermediate memory component 320 , a memory management component 314 , and a hypervisor 312 .

As shown in FIG. 3 , computing system 301 may include a persistent memory device 316 in which logic circuitry 304 resides. Persistent memory device 316 may include multiple arrays of memory cells, such as array 322 . The array 322 may be coupled to the logic circuitry 304 via a data bus located within the persistent memory device 316 . As described herein, array 322 may include an array of resistive memory cells, a phase change memory device, an array of self-selecting memory cells, or other suitable non-volatile memory resource, or combinations thereof.

In some embodiments, computing system 301 may be a multi-user network, such as a software defined data center, cloud computing environment, or the like. In such embodiments, the computing system may be configured to execute one or more virtual machines 317 . For example, in some embodiments, one or more virtual machines 317 may be deployed on hypervisor 312 and accessed by users of a multi-user network.

I/O device 310 , persistent memory device 316 , non-persistent memory device 330 , intermediate memory component 320 , and memory management component 314 are I/O device 210 shown in FIG. 2 . , persistent memory device 216 , non-persistent memory device 230 , intermediate memory component 220 , and memory management component 214 . Communication between I/O device 310 and persistent memory device 316 (as well as logic circuitry 304 ), non-persistent memory device 330 , hypervisor 312 , and memory management component 314 is illustrated in FIG. This may be enabled via interface 308 , which may be similar to interface 208 shown in FIG. 2 .

As described above with respect to FIG. 2 , the memory management component 314 may cause a read request or a write request associated with the I/O device 310 to be redirected to the logic circuitry 304 . The logic circuitry 304 may generate and/or store a logical address corresponding to the requested data. As described above, the logic circuitry 304 may store a logical address corresponding to the requested data in a base address register, such as the address register 306-N of the logic circuitry 304 .

As shown in FIG. 3 , hypervisor 312 may communicate with logic circuitry 304 and/or I/O devices 310 via interface 308 . Hypervisor 312 may also communicate with persistent memory device 316 (as well as logic circuitry 304 ), non-persistent memory device 330 , intermediate memory component 320 , and memory management component 314 . have. The hypervisor may be configured to execute special instructions to perform the operations and/or tasks described herein. For example, the hypervisor 312 may allow data to be transferred via the interface 308 while data is transferred between the array 222 and logic circuitry 204 via a data bus 218 within the persistent memory device 216 . It may execute instructions to be transmitted between the logic circuitry 304 and the I/O device 310 .

Hypervisor 312 also executes instructions to monitor data traffic and data traffic patterns to determine whether data should be stored in non-persistent memory device 330 or data should be transferred to persistent memory device 316 . can That is, in some embodiments, the hypervisor 312 learns user data request patterns over time and based on those patterns portions of data to the non-persistent memory device 330 or persistent memory device 316 . You can execute commands to selectively save to . This may allow more frequently accessed data to be stored in non-persistent memory device 330 , while less frequently accessed data may be stored in persistent memory device 316 .

Because the user may access recently used or viewed data more frequently than less recently used or less recently viewed data, the hypervisor causes less recently used or viewed data to be stored in persistent memory device 316 and /or execute special instructions that cause recently more accessed or viewed data to be stored in the non-persistent memory device 330 . In a non-limiting example, a user may view photos on social media that they have taken recently (eg, within a week, etc.) more frequently than photos they have taken less recently (eg, a month ago, a year ago, etc.). can see. Based on this information, hypervisor 312 may execute special instructions to cause less recently viewed or taken pictures to be stored on persistent memory device 316 , thereby causing non-persistent memory device 330 to store them. It can reduce the amount of data. This may reduce the overall amount of non-persistent memory required to provision the computing system 301 , thereby reducing cost and allowing access to the non-persistent memory device 330 to more users.

In operation, computing system 301 may be configured to intercept data requests from I/O devices 310 and redirect the requests to logic circuitry 304 . In some embodiments, hypervisor 312 may control whether data corresponding to a data request is to be stored in (or retrieved from) non-persistent memory device 330 or stored in persistent memory device 316 . can For example, hypervisor 312 may be configured to selectively control whether data is stored to (or retrieved from) persistent memory device 316 or non-persistent memory device 330 (or retrieved from). commands can be executed.

As part of controlling whether data is stored (or retrieved) to the persistent memory device 316 and/or stored (or retrieved) to the non-persistent memory device 330 , the hypervisor 312 provides the memory management component 314 . ) to map the logical addresses associated with the data to be redirected to the logic circuitry 304 and stored in the address registers 306 of the logic circuitry 304 . For example, hypervisor 312 may execute instructions to control read and write requests that involve data being selectively redirected to logic circuitry 304 via memory management component 314 .

Memory management component 314 may map contiguous virtual addresses to underlying fragmented physical addresses. Accordingly, in some embodiments, memory management component 314 may enable virtual addresses to be mapped to physical addresses without the requirement that the physical addresses are concatenated. Furthermore, in some embodiments, the memory management component 314 can allow devices that do not support memory addresses long enough to address their corresponding physical memory space to be addressed to the memory management component 314 . .

Due to the non-deterministic nature of data transfers associated with persistent memory device 316 , logic circuitry 304 may, in some embodiments, result in delays in transferring data to or from persistent memory device 316 . and notify the computing system 301 that there is. As part of initiating the delay, logic circuitry 304 may provide page fault handling for computing system 301 when a data request is redirected to logic circuitry 304 . In some embodiments, logic circuitry 304 may generate an interrupt and assert it to hypervisor 312 to initiate an operation to transfer data into or out of persistent memory device 316 . For example, due to the non-deterministic nature of data retrieval and storage associated with the persistent memory device 316 , the logic circuitry 304 may cause a hypervisor interrupt ( 315) can be created.

In response to the page fault interrupt generated by the logic circuitry 304 , the hypervisor 312 may retrieve information corresponding to the data from the logic circuitry 304 . For example, hypervisor 312 may receive NIC access data from logic circuitry that may include logical-to-physical address mappings corresponding to data stored in address registers 306 of logic circuitry 304 .

Once the data has been stored in the persistent memory device 316 , a portion (eg, pages, blocks, etc.) of the non-persistent memory device 330 allows the computing system 301 to access the data from the non-persistent memory device 330 . may be marked as inaccessible by logic circuitry 304 so as not to attempt to do so. This is a data request by a page fault generated by the logic circuitry 304 and asserted to the hypervisor 312 when data that was stored in the persistent memory device 316 is requested by the I/O device 310 . This can be made to be intercepted.

In contrast to approaches in which a page fault exception is raised in response to an application requesting access to a page of memory that is not mapped by a memory management unit (eg, memory management component 314 ), an embodiment of the present disclosure In the above described page faults, the logic circuitry 304 in response to data being mapped to the logic circuitry 304 in the memory management component 314 , which in turn maps the data to the persistent memory device 316 . ) can be created by

In some embodiments, the intermediate memory component 320 may be used to buffer data that is stored in the persistent memory device 316 in response to a data request initiated by the I/O device 310 . In contrast to persistent memory device 316 , which may transfer data via a PCIe interface, intermediate memory component 320 may employ a DDR interface to transfer data. Accordingly, in some embodiments, the intermediate memory component 320 may operate in a deterministic manner. For example, in some embodiments, the requested data stored in the persistent memory device 316 may be temporarily transferred from the persistent memory device 316 to the intermediate memory component 320 , followed by the intermediate memory component ( may be sent to the host computing device via a DDR interface coupling 320 to I/O device 310 .

In some embodiments, the intermediate memory component may include a separate memory component (eg, SRAM cache) disposed in the computing system 301 . However, embodiments are not so limited, and in some embodiments, the intermediate memory component 320 is a non-persistent memory that may be allocated for use in transferring data from the persistent memory device 316 in response to a data request. It may be part of device 330 .

In a non-limiting example, logic circuitry (eg, logic circuitry 304 ) may reside on a permanent memory device (eg, permanent memory device 316 ). The logic circuitry may include an address register configured to store logical addresses corresponding to data stored in the persistent memory device. The logic circuitry may be configured to receive the redirected request to retrieve the portion of data stored in the persistent memory device. Prior to the redirect, the request is directed to the non-persistent memory device. The logic circuitry may be configured to, in response to receiving a request to retrieve the portion of data stored in the persistent memory device, determine a physical address corresponding to the portion of data based on the logical address stored in the address register. The logic circuitry may be configured to cause data to be retrieved from the persistent memory device based on the determined address.

In some embodiments, the logic circuitry may be configured to generate, based on the determined address, a request to retrieve the portion of data stored from the persistent memory device. The logic circuitry may also be configured to send a request (eg, a previously generated request) to the hypervisor via an interface that couples the logic circuitry to the hypervisor. In response to receiving the request at the hypervisor, the hypervisor may be configured to enable retrieval of the portion of data. The logic circuitry may be configured to generate the interrupt signal and assert the interrupt signal to the hypervisor as part of a request to retrieve, via the interface, the portion of data from the persistent memory device. In response to sending the request to the hypervisor, the logic circuitry may be configured to receive the portion of data from the persistent memory device and transmit the portion of data to the input/output (I/O) device via the interface.

In some embodiments, the persistent memory device may be one and/or a combination of various non-volatile memory resources. For example, a permanent memory device may include an array of resistive memory cells, a phase change memory device, an array of self-selecting memory cells, or other suitable non-volatile memory resource, or combinations thereof.

In some embodiments, the logic circuitry may be configured to associate, based on receipt of the request, an indication indicating that the portion of data is inaccessible to the non-persistent memory device with the portion of data. The logic circuitry may include a state machine.

In another non-limiting example, a system can include a permanent memory device having logic circuitry in which it resides and memory management circuitry coupled to the permanent memory device and the logic circuitry. The logic circuitry may be configured to, in response to receiving the redirected request from the memory management circuitry to write data to the persistent memory device, send a request to the hypervisor to write data to the persistent memory device. The logic circuitry may also be configured such that the hypervisor causes data associated with the request to be written from the logic circuitry to the persistent memory device. The logic circuitry may include a buffer configured to store data to be written to the permanent memory device (eg, write buffer 105 shown in FIG. 1 herein).

In some embodiments, the request is received at the logic circuitry via an interface that couples the logic circuitry to the memory management circuitry. In contrast (eg, that an interface was used to receive a request from the logic circuitry), the logic circuitry may be configured to cause data to be written from the logic circuitry to the persistent memory device via a different data bus than the interface.

In some embodiments, the logic circuitry is configured to receive the redirected request from the memory management circuitry to retrieve data from the persistent memory device. In response to receiving the request, the logic circuitry may be configured to send a request to the hypervisor to retrieve data from the persistent memory device and assert an interrupt signal to the hypervisor as part of the transmitted request.

In response to receiving the transmitted data, the hypervisor may be configured to cause the data to be retrieved from the persistent memory device and transferred to the non-persistent memory device via an interface coupling the persistent memory device to the non-persistent memory device. Further, in response to receiving the retrieved data from the persistent memory device, the logic circuitry may be configured to transmit the data to the I/O device via an interface coupling the persistent memory device to an input/output (I/O) device. .

In some embodiments, the logic circuitry receives virtual I/O device access information from an input/output (I/O) device and virtual I/O device access information as part of a request to write data to a persistent memory device. may be configured to send to the hypervisor. The logic circuitry may include an array of memory cells configured to store virtual I/O device access information.

4 is a flowchart 440 illustrating a data read operation in accordance with various embodiments of the present disclosure. At block 441, an I/O device such as I/O device 210/310 shown in FIGS. 2 and 3 may initiate a read operation using the address corresponding to the data request. In some embodiments, the address may be a physical address, such as a virtual machine physical address. A data request may include a request to read data associated with a particular address corresponding to a logical address at which data is stored. A physical address may be a location in a persistent memory device (eg, the persistent memory device 216/316 shown in FIGS. 2 and 3 herein) or a non-persistent memory (eg, herein, FIGS. 2 and 3 ). 3 may correspond to a location in the non-persistent memory 230/330).

If the data is stored in the non-persistent memory device, the data may be retrieved and the data request may be fulfilled. However, if the data is stored in the persistent memory device (eg, the physical address of the data corresponds to a location in the persistent memory device), then at block 442 a memory management component (eg, herein, The memory management component 214/314 shown in FIGS. 2 and 3 ) sends data requests to logic circuitry (eg, logic circuitry 104/204/304 shown in FIGS. 1-3 herein). can be redirected As described above, a data request may be redirected based on information (eg, a command or instructions being executed) by the hypervisor (eg, the hypervisor 312 shown in FIG. 3 herein). have.

At block 443 , the logic circuitry may receive address register access information corresponding to the data request. In some embodiments, the address register access information may correspond to a location in an address register (eg, address registers 106/206/306 shown in FIGS. 1-3 herein). For example, the address register access information may correspond to a location in an address register within the logical circuitry where a logical address corresponds to a physical address within a persistent memory device in which data is stored.

The logic circuitry may generate a hypervisor interrupt at block 444 . For example, as described above with respect to FIG. 3 , if the logic circuitry has received a redirected data request from the memory management component, the logic circuitry generates an interrupt and the hypervisor (eg, herein, in FIG. 3 ) generates an interrupt. It may assert an interrupt on the hypervisor 312 shown. In some embodiments, an interrupt may be a signal asserted on the hypervisor to notify the hypervisor that an event requires immediate attention. For example, an interrupt signal may be asserted on the hypervisor to cause the hypervisor to interrupt instructions currently being executed and instead execute instructions associated with gathering address register access information at block 445 . .

At block 445 , the hypervisor may gather address register access information from the logic circuitry. For example, the hypervisor may receive logical address information corresponding to the physical address of the requested data from the logical circuit unit. Logical address information may be stored herein in logic circuitry within address registers (eg, base address registers) of logical circuitry, such as address register(s) 106/206/306 shown in FIGS. have.

At block 446 , the hypervisor may determine the physical location of the requested data. For example, based on the address register access information, and hence the logical address associated with the data collected from the logical circuitry, the hypervisor may determine the physical location of the data stored in the persistent memory device.

At block 447 , the hypervisor may read data corresponding to the address register access information. That is, in some embodiments, the hypervisor may cause the requested data to be read (eg, retrieved) from the persistent memory device.

At block 448 , the hypervisor may cause the data to be transferred to the non-persistent memory device. In some embodiments, the non-persistent memory device may be the non-persistent memory device 230/330 shown in FIGS. 2 and 3 herein.

At block 449, the hypervisor may write the I/O device data corresponding to the requested data to the logic circuitry. I/O device data may be stored in address registers of the logic circuit portion, as described above. In some embodiments, the hypervisor also sends data from the logic circuitry to an intermediate memory component, such as intermediate memory component 220/320 shown in FIGS. /O Can transmit device data.

At block 450, the logic circuitry may complete the data read transaction. For example, the logic circuitry may send a command to the I/O device to notify the I/O device that the data read request has been fulfilled and that data will be sent over the deterministic interface to fulfill the data read request.

At block 451 , the hypervisor may update the memory management component to direct the I/O device address to the non-persistent memory device. For example, at block 450 , since the data has been transferred from the persistent memory device to the non-persistent memory device (eg, the non-persistent memory device and/or intermediate memory component), the hypervisor sends an address corresponding to the requested data. may update the memory management component to map to a non-persistent memory device. In some embodiments, the address may be physical addresses, such as a virtual machine physical address.

At block 452, the hypervisor may record which memory was used to satisfy the data request. For example, the hypervisor may record that the data was stored in the persistent memory device at the time the data request was received from the I/O device. In some embodiments, the hypervisor may use the information over time to selectively direct data writes to a persistent memory device or a non-persistent memory device.

5 is a flowchart 560 illustrating a data write operation in accordance with various embodiments of the present disclosure. At block 561, an I/O device such as I/O device 210/310 shown in FIGS. 2 and 3 may initiate a write operation using the address corresponding to the data request. The address may be a physical address, such as a virtual machine physical address. A data write request may include a request to write data associated with a particular virtual address corresponding to a logical address at which the data is to be stored. A physical address may be a location in a persistent memory device (eg, the persistent memory device 216/316 shown in FIGS. 2 and 3 herein) or a non-persistent memory (eg, herein, FIGS. 2 and 3 ). may correspond to a location in non-persistent memory 230/330 shown in 3. For example, a physical address may correspond to a number of arrays of memory cells of a persistent memory device (eg, herein, FIGS. 2 and 2 and FIG. 3) may correspond to the arrays 222/322).

When the data is stored in the non-persistent memory device, the data may be written to the non-persistent memory device, and the data write request may be fulfilled. However, if the data is to be stored in the persistent memory device, then at block 422, a memory management component (eg, the memory management component 214/314 shown in FIGS. 2 and 3 herein) sends a data write request. through an interface (eg, interface 208/308 shown in FIGS. 2 and 3) via logic circuitry (eg, herein, logic circuitry 104/204/304 shown in FIGS. )) can be redirected. As described above, a data request may be redirected based on information (eg, a command or instructions being executed) by the hypervisor (eg, the hypervisor 312 shown in FIG. 3 herein). have.

At block 563, the logic circuitry may receive address register access information corresponding to the data write request. In some embodiments, the address register access information may correspond to a location in an address register (eg, address registers 106/206/306 shown in FIGS. 1-3 herein). For example, the address register access information may correspond to a location in an address register within the logical circuitry whose logical address corresponds to a physical address within the permanent memory device in which the data is to be stored.

The logic circuitry may generate a hypervisor interrupt at block 564 . For example, as described above with respect to FIG. 3 , if logic circuitry has received a redirected data write request from a memory management component, the logic circuitry generates an interrupt and triggers the hypervisor (eg, herein, FIG. 3 ) It is possible to assert an interrupt on the hypervisor 312 shown in FIG.

At block 565 , the hypervisor may gather address register access information from the logic circuitry. For example, the hypervisor may receive logical address information corresponding to a physical address in which data is to be stored from the logical circuit unit.

At block 566, the hypervisor may optionally write (or cause the data to be written) to the persistent memory device. For example, based on a redirected data write request, the hypervisor may may determine to be written to the persistent memory device and cause the data to be written to the persistent memory In embodiments where block 566 is optionally performed, the data may be intermittently written to the non-persistent memory device. Also, I/O device data corresponding to the data may optionally be written to the non-persistent memory device as part of writing the data to the non-persistent memory device.

Optionally, at block 567, the hypervisor may write (or cause data to be written to) the non-persistent memory device. In some embodiments, the hypervisor may request a read corresponding to the data. may write data to a non-persistent memory device such that when received, the data can be retrieved via a deterministic interface or bus.

At block 568 , the hypervisor may update the memory management component to direct the I/O device virtual addresses to the non-persistent memory device. For example, if data is written to the non-persistent memory device at block 567 , then at blocks 568 the hypervisor associates the data with the virtual addresses stored by the memory management component to the non-persistent memory device where the data is stored. may update the virtual addresses stored by the memory management component to be mapped to physical addresses in

6 is a flow diagram illustrating an exemplary method 670 for logic circuitry in a memory in accordance with various embodiments of the present disclosure. At block 672 , the method 670 makes a redirected request to retrieve data from the persistent memory device via an interface (eg, interface 208/308 shown herein in FIGS. 2 and 3 ). Logic circuitry (eg, herein, shown in FIGS. receiving at the logic circuitry 104/204/304). The request may be directed to the original non-persistent memory device (eg, non-persistent memory device 230/330 shown in FIGS. 2 and 3 herein) prior to the redirect.

At block 674 , the method 670 sends a request to retrieve data from the persistent memory device to a hypervisor coupled to the logic circuitry (eg, the hypervisor 312 shown in FIG. 3 herein). may include the step of At block 676 , the method 670 may include transferring data retrieved from the persistent memory device to logic circuitry or to a non-persistent memory device (eg, a non-persistent memory device via an interface), or both. can Data from the permanent memory device (eg, an array of memory cells of the permanent memory device) to the logic circuitry may be retrieved via a data bus that couples the permanent memory device (eg, an array thereof) to the logic circuitry.

The data retrieved to the hypervisor may be passed to various components of the hierarchical memory system. For example, in some embodiments, data retrieved from a persistent memory device is transferred to an input/output (I/O) device (eg, herein, FIG. 2 ) via an interface that couples logic circuitry to the I/O device. and I/O devices 210/310 shown in FIG. 3).

In some embodiments, method 670 associates, by logic circuitry, with the data an indication indicating that the data is inaccessible to the non-persistent memory device based on receipt of a request to retrieve data from the persistent memory device. It may further include a step of making. In some embodiments, method 670 includes generating, by logic circuitry, a logical address corresponding to the data in response to receiving the request; and storing the logical address in an address register in the logical circuit unit.

Although specific embodiments have been illustrated and described herein, it will be understood by those skilled in the art that arrangements calculated to achieve the same results may be substituted for the specific embodiments shown. This disclosure is intended to cover adaptations or variations of one or more embodiments of the disclosure. It is to be understood that the foregoing description has been made in an illustrative and not a restrictive manner. Combinations of the above-described embodiments and other embodiments not specifically described herein will become apparent to those skilled in the art upon review of the foregoing description. The scope of one or more embodiments of the present disclosure includes other applications in which the structures and processes described above are used. Accordingly, the scope of one or more embodiments of the present disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are granted.

In the foregoing detailed description, some features are grouped together in one embodiment to simplify the present disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the disclosed embodiments of the present disclosure may employ more features than are expressly recited in each claim. More precisely, as the following claims reflect, inventive subject matter lies in less than all features of one disclosed embodiment. Accordingly, the following claims are hereby incorporated into the detailed description for carrying out the invention, each claim being a separate embodiment and stand alone.

Claims (20)

An apparatus comprising: a persistent memory device; and logic circuitry resident on the permanent memory device and comprising an address register configured to store logical addresses corresponding to data stored in the permanent memory device, wherein the logic circuitry includes: the data stored in the permanent memory device to receive a redirected request to retrieve a portion of , wherein, prior to the redirect, the request is directed to a non-persistent memory device; in response to receiving the request to retrieve the portion of data stored in the persistent memory device, determine a physical address corresponding to the portion of data based on the logical address stored in the address register; and based on the determined physical address, cause the data to be retrieved from the persistent memory device. 2. The device of claim 1, wherein the logic circuitry is configured to: generate, based on the determined physical address, a request to retrieve the stored portion of the data from the persistent memory device; send the request to the hypervisor via an interface coupling the logic circuitry to the hypervisor; and the hypervisor is configured to facilitate retrieval of the portion of data in response to receiving the request at the hypervisor. 3. The method of claim 2, wherein the logic circuitry is configured to: in response to sending the request to the hypervisor, receive the portion of the data from the persistent memory device; and transmit the portion of data to an input/output (I/O) device via the interface. 3. The method of claim 2, wherein the logic circuitry is configured to: generate an interrupt signal; and
and assert, via the interface, the interrupt signal to the hypervisor as part of the request to retrieve the portion of data from the persistent memory device. The apparatus of claim 1 , wherein the permanent memory device comprises an array of resistive memory cells, a phase change memory device, an array of self-selecting memory cells, or combinations thereof. The apparatus of claim 1 , wherein the logic circuitry is configured to associate, based on receipt of the request, an indication indicating that the portion of data is inaccessible to the non-persistent memory device with the portion of data. 2. The apparatus of claim 1, wherein the logic circuitry comprises a state machine. A system comprising: a permanent memory device having resident logic circuitry; and
memory management circuitry coupled to said persistent memory device and said logic circuitry; the logic circuitry is configured to: in response to receiving a request redirected from the memory management circuitry to write data to the persistent memory device, send a request to write the data to the persistent memory device to a hypervisor; the hypervisor is configured to cause the data associated with the request to be written from the logic circuitry to the persistent memory device; the request is received by the logic circuitry through an interface coupling the logic circuitry to the memory management circuitry; and the logic circuitry is configured to cause the data to be written from the logic circuitry to the permanent memory device via a different data bus than the interface. 9. The system of claim 8, wherein the logic circuitry includes a buffer configured to store the data to be written to the permanent memory device. A method, comprising: receiving, at logic circuitry residing on a persistent memory device, a redirected request to retrieve data from the persistent memory device via an interface, prior to the redirect, the request is directed to a non-persistent memory device; sending a request to retrieve the data from the persistent memory device to a hypervisor coupled to the logic circuitry; and transferring the data retrieved from the persistent memory device to the logic circuitry or the non-persistent memory device, or both, via the interface; prior to transferring the data via the interface, causing the data to be retrieved from the persistent memory device to the logic circuitry via a data bus coupling the persistent memory device to the logic circuitry. A system comprising: a permanent memory device having resident logic circuitry; and memory management circuitry coupled to the permanent memory device and the logic circuitry; the logic circuitry is configured to, in response to receiving a redirected request from the memory management circuitry to write data to the persistent memory device, send a request to write the data to the persistent memory device to a hypervisor; the hypervisor causes the data associated with the request to be written from the logic circuitry to the persistent memory device; receive a redirected request from the memory management circuitry to retrieve the data from the persistent memory device; send a request to the hypervisor to retrieve the data from the persistent memory device; and assert an interrupt signal to the hypervisor as part of the transmitted request. 12. The non-persistent memory of claim 11, wherein the hypervisor, in response to receiving the transmitted request, causes the data to be retrieved from the persistent memory device and via an interface coupling the persistent memory device to a non-persistent memory device. and the system is configured to be transmitted to the device. 12. The I/O device of claim 11, wherein the logic circuitry is responsive to receiving the data retrieved from the persistent memory device, via an interface coupling the persistent memory device to an input/output (I/O) device. and transmit the data to A system comprising: a permanent memory device having resident logic circuitry; and memory management circuitry coupled to the permanent memory device and the logic circuitry; the logic circuitry is configured to, in response to receiving a redirected request from the memory management circuitry to write data to the persistent memory device, send a request to write the data to the persistent memory device to a hypervisor, the The hypervisor is configured to cause the data associated with the request to be written from the logic circuitry to the persistent memory device, the logic circuitry configured to: receive, from an input/output (I/O) device, virtual I/O device access information so; and send the virtual I/O device access information to the hypervisor as part of the request to write the data to the persistent memory device. 15. The system of claim 14, wherein the logic circuitry comprises an array of memory cells configured to store the virtual I/O device access information. A method, comprising: receiving, at logic circuitry residing on a persistent memory device, a redirected request to retrieve data from the persistent memory device via an interface, prior to the redirect, the request is directed to a non-persistent memory device; sending a request to retrieve the data from the persistent memory device to a hypervisor coupled to the logic circuitry; transferring the data retrieved from the persistent memory device to the logic circuitry or the non-persistent memory device, or both, via the interface; and transferring the data retrieved from the persistent memory device to the I/O device via the interface coupling the logic circuitry to an input/output (I/O) device. A method, comprising: receiving, at logic circuitry residing on a persistent memory device, a redirected request to retrieve data from the persistent memory device via an interface, prior to the redirect, the request is directed to a non-persistent memory device; sending a request to retrieve the data from the persistent memory device to a hypervisor coupled to the logic circuitry; transferring the data retrieved from the persistent memory device to the logic circuitry or the non-persistent memory device, or both, via the interface; and associating, by the logic circuitry, with the data an indication indicating that the data is inaccessible to the non-persistent memory device based on receipt of the request to retrieve the data from the persistent memory device. , Way. A method, comprising: receiving, at logic circuitry residing on a persistent memory device, a redirected request to retrieve data from the persistent memory device via an interface, prior to the redirect, the request is directed to a non-persistent memory device; sending a request to retrieve the data from the persistent memory device to a hypervisor coupled to the logic circuitry; transferring the data retrieved from the persistent memory device to the logic circuitry or the non-persistent memory device, or both, via the interface; generating, by the logic circuitry, a logical address corresponding to the data in response to receiving the request; and storing the logical address in an address register within the logic circuitry. delete delete

Images