WO2016173611A1

WO2016173611A1 - Memory systems

Info

Publication number: WO2016173611A1
Application number: PCT/EP2015/059047
Authority: WO
Inventors: Luis Miguel Vaquero Gonzalez; Suksant SAE LOR
Original assignee: Hewlett-Packard Development Company, L P
Priority date: 2015-04-27
Filing date: 2015-04-27
Publication date: 2016-11-03
Also published as: TW201706859A

Abstract

A memory system is described which comprises a plurality of memory modules. Each memory module comprises a non-volatile memory unit; and a controller to access information stored in the non-volatile memory unit in response to a request from a processor, to deliver accessed information to the processor, and to communicate with controllers of other memory modules. The controllers are connected so as to form a peer-to-peer network. Each given memory module is associated with a memory address range which is non-overlapping with a memory address range associated with any memory module to which the given memory module is directly connected.

Description

MEMORY SYSTEMS

BACKGROUND

[0001] Non-volatile memory is computer memory that can hold stored information even when not-powered. Computer systems use non-volatile memory (NVM) for long-term persistent storage. In some systems non-volatile memory is provided remotely from the processor using it, e.g. in a remote data centre, enabling a larger storage capacity than is possible by exclusively using memory local to the processor.

BRIEF DESCRIPTION OF DRAWINGS

[0002] Examples will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which:

[0003] Figure 1 is a simplified schematic of an example memory system;

[0004] Figure 2 illustrates an example routing protocol used by an example memory system;

[0005] Figure 3 is a flowchart of an example of a method of joining a controller to the peer-to-peer network of an example memory system; and

[0006] Figure 4 is a flowchart of an example method for accessing information stored in an example memory system.

DETAILED DESCRIPTION

[0007] Many computing tasks involve the processing of massive amounts of data. To deal with such massive data sets, processors having multiple cores and/or linked networks of processors can be used, and data can be stored in memory resources located remotely to where the processing is performed. However; such arrangements involve frequent transfers of data between remote locations. The time involved in accessing remote memory resources counteracts increases in processor speed, making it difficult to reduce the time overhead of data processing.

[0008] Recent advances in photonics have enabled optical connections to external memory resources. Such optical interconnects provide reduced latencies and potentially higher bandwidths, making accesses to remote memory resources nearly as fast as local accesses. Thus, the possibility of a "memory fabric" which integrates multiple different memory resources, across several racks of a data centre (or even across several data centres) is created.

[0009] However, memory resources from different vendors may use a variety of different physical and virtual memory protocols. A memory fabric comprising memory resources from different vendors can therefore include a set of protocols that make processor-memory communication possible across all of the vendors and solutions encompassed in that memory fabric.

[0010] It can be envisaged that changes to the make-up of a given memory fabric may be effected, for example to add or remove memory resources, or in case of failure of memory resources comprised in the fabric. Furthermore, management, utilization and access characteristics may change dynamically. Depending on the number of cores being used and the application at hand, the best fitting memory for a given processor may change or may be unavailable at the moment of scheduling of the application. A challenge to be addressed in the design of a memory fabric is therefore the provision of a memory management system which is highly flexible such that it can react to changing system states of the memory resources in the memory fabric and changing run-time behaviour of processors accessing the memory fabric.

[0011] A "plug and play" device is a peripheral computer component having a specification which, on connection of the device to a computer system, allows it to be automatically discovered and installed by the computer system. A plug and play device can be often be connected to and disconnected from a computer system while the computer system is operating.

[0012] Examples of a system enabling a non-volatile memory component to be dynamically added to a memory system, such as a memory fabric, in a manner akin to the connection of a plug and play device to a computer system are described in the following. The described examples provide a NVM system in which the configuration of the memory system is not bound in any way to a processor accessing the memory. This means that a memory component can be unplugged from one memory system and connected to a different memory system, where it will operate and integrate just as it did when connected to the original memory system.

[0013] Figure 1 shows an example of a memory system 1 . The memory system 1 comprises a plurality of memory modules 10. In this example the memory modules 10 are distributed between a first memory component 2 and a second memory component 3. The first memory component 2 and the second memory component 3 may (but need not) be located remotely from each other. Each memory module 10 comprises a non-volatile memory (NVM) unit and a controller. Each controller is to access information stored in the non-volatile memory unit in response to a request from a processor, to deliver accessed information to the processor, and to communicate with controllers of other memory modules. The controllers are connected so as to form a peer-to-peer network. The controllers thereby comprise peers in a peer-to-peer network.

[0014] Linking the memory module controllers as peers in a peer-to-peer network enables the memory system to autonomously discover and configure new memory modules of the system (i.e. memory modules which have recently been provided in communication with at least one controller in the peer-to-peer network) in a decentralized manner. Consequently, the memory system is scalable and can tolerate the failure of individual memory modules. Furthermore, example memory systems need not use wired interconnects or memory switches. The combination of these features makes it possible to federate memory spaces of otherwise independent NVM units (i.e. NVM units which comprise separate physical devices, and/or are in different geographic locations, and/or have no connections to each other). The federated memory space created by the example memory systems appears to an external processor to be a single memory space, even as the memory modules comprised in the system change.

[0015] Each memory module is associated with a range of memory addresses. In some examples each memory module is associated with a unique range of memory addresses. These memory address ranges are dynamically assigned in dependence on an addressing schema enabled by the underlying protocol used by the memory system. In some examples the range of memory addresses associated with a given memory module is determined during joining of the controller of that memory module to the peer-to-peer network. In some examples the controller of a given memory module is to determine a memory address range for that memory module when the memory module is initially connected to a memory system (this process can be seen as akin to finding an available subrange in an IP network). In some such examples the controller is to find available memory address ranges by contacting neighbouring controllers (i.e. controllers which are directly connected to the newly joined controller) and obtaining their memory maps, so that available ranges of memory addresses (i.e. ranges of memory addresses not used by any neighbouring controllers) can be determined.

[0016] The controller of a given memory module may advertise the memory address range associated with (and therefore managed by) that memory module. In some examples the controller of a given memory module is to advertise the capabilities of that memory module. In some such examples the advertised capabilities include any or all of: size (i.e. memory capacity), technology, installation date, faulty addresses, etc. [0017] Each controller is associated with a role, where the role is one of: router, switch, and plain memory. A controller associated with the role of router will hereinafter be referred to as a router controller, a controller associated with the role of switch will hereinafter be referred to as a switch controller, and a controller associated with the role of plain memory will hereinafter be referred to as a plain memory controller. An example memory system 1 comprises thirteen plain memory controllers (denoted by the letter "M" in Figure 1 ), three switch controllers (denoted by the letter "S" in Figure 1 ) and two router controllers (denoted by the letter " " in Figure 1 ). Each memory component 2, 3 comprises a single router controller R.

[0018] Each router controller stores a routing table comprising an identifier for every other router controller in the peer-to-peer network and is to route memory access requests (e.g. originating from a processor) according to the stored routing table. Each switch controller is to forward memory access requests to every other controller in the peer-to-peer network. Like in an IP network, in some examples, some networked memory modules can forward memory access requests to all the connected memory modules (switching), and some networked memory modules can route memory access requests based on routing tables (in a similar manner to routers in IP networks). Plain memory controllers may not perform routing or switching functions.

[0019] Each router controller is associated with a unique identifier. In some examples, an operator of the memory system manually assigns the identifiers to the router controllers. In some examples, the assignment of identifiers to router controllers is performed by a management layer of the memory system. In some examples, the peer-to-peer network stores the identifiers in a distributed hash table (DHT). In some examples the identifiers are uniformly distributed in an identifier space. A uniform distribution can be achieved, for example, by setting the identifier of a given router controller to be the cryptographic hash of a vendor-assigned component ID (e.g. the vendor-assigned component ID of the NVM unit with which the router controller is associated).

[0020] In some examples the identifiers are ordered in an identifier ring, such that each controller has a predecessor and a successor in the identifier ring. In some examples each controller can periodically send keep-alive messages to its predecessor and its successor, e.g. in accordance with a routing protocol of the memory system.

[0021] A variety of routing protocols could be used by the example memory system 1 . An example of a suitable routing protocol will now be described with reference to Figure 2.

[0022] The example routing protocol comprises a one-hop scheme in which all of the router controllers 20 of a memory system (e.g. the memory system 1 of Figure 1 ) form a ring and every router controller 20 maintains a full routing table containing information about every other router controller 20 in the ring. The information in each of the routing tables may comprise, for example, any or all of: identifier, location in the ring, memory ranges, size, a summary of technologies and faulty components, IDs of controllers (of all roles) to which it is directly connected, etc.

[0023] Each router controller 20 is associated with a random 128-bit component identifier and the identifiers are ordered in an identifier ring modulo 2^Λ128.

[0024] The query success rate of the memory system depends on the accuracy of the information in the routing tables of the router controllers 20. To maintain correct local information (i.e., information about its successor and predecessor), each given router controller n periodically runs a stabilisation routine, wherein it sends keep-alive messages to its successor s and predecessor p. On receipt of such a keep-alive message, the successor s checks if n is indeed its predecessor (e.g. by comparing the identifier of n to the predecessor identifier stored in its routing table) and if not, successor s notifies n of the existence of another router controller between them. Similarly, on receipt of a keep-alive message from n, p checks if n is indeed its successor (e.g. by comparing the identifier of n to the successor identifier stored in its routing table), and if not it notifies n. If either of s or p does not respond to a keep-alive message, n pings the non-responsive component repeatedly until a time-out period has expired. On the expiry of the time-out period, n decides that the non-responsive component is unreachable or dead.

[0025] To maintain correct full routing tables responsive to controllers joining or leaving the peer-to-peer network (e.g. because a memory component is added or removed from the memory system), notifications of network membership change events (i.e., joins and leaves) should reach every router controller 20 in the network within a specified amount of time. Such notifications may originate from a controller which is directly connected to the controller of a newly added memory module, or which was formerly (i.e. prior to the removal) directly connected to the controller of a newly removed memory module. In some examples the specified time is based on network size (i.e. the number of router controllers in the identifier ring), and may be less than one second. The example routing protocol achieves this in the following manner.

[0026] The 128-bit circular identifier space is divided into k equal contiguous intervals, hereinafter referred to as slices, k may be selected by an operator of the memory system, k may be selected, for example, based on any or all of: network topology, users being served, level of availability sufficient to prevent total failure, etc. In a data centre, for instance, k may be selected such that there are no router controllers from the same region of the data centre in the same slice. In the example of Figure 2 the identifier ring is divided into two slices (by the dashed line A), so k = 2. The slice contains all of the rou currently in the peer-to-peer network whose identifiers lie in the range i - . Since the

router controllers 20 have uniformly distributed random identifiers, each slice will comprise about the same number of router controllers 20 as each other slice, at any given time.

[[00002277]] TThhee rroouuttiinngg pprroottooccooll ddeeffiinneess aa sslliiccee lleeaaddeerr 2211 ffoorr eeaacchh sslliiccee.. TThhee sslliiccee lleeaaddeerr iiss ddeeffiinneedd ddyynnaammiiccaallllyy aass tthhee rroouutteerr ccoonnttrroolllleerr hhaavviinngg aann iiddeennttiiffiieerr tthhaatt iiss tthhee ssuucccceessssoorr ooff tthhee iiddeennttiiffiieerr aatt tthhee mmiiddppooiinntt ooff tthhee sslliiccee rraannggee.. IIff aa sslliiccee ccoommpprriisseess aann eevveenn nnuummbbeerr ooff iiddeennttiiffiieerrss,, ssuucchh tthhaatt tthheerree iiss nnoo iiddeennttiiffiieerr ccoorrrreessppoonnddiinngg ttoo aann eexxaacctt mmiiddppooiinntt,, aa rroouutteerr ccoonnttrroolllleerr hhaavviinngg aann iiddeennttiiffiieerr cclloosseesstt ttoo tthhee mmiiddppooiinntt iiss ddeeffiinneedd ttoo bbee tthhee sslliiccee lleeaaddeerr.. FFoorr eexxaammppllee,, tthhee sslliiccee lleeaaddeerr ooff tthhee ^hh sslliiccee iiss tthhee ssuucccceessssoorr ooff tthhee rroouutteerr ccoonnttrroolllleerr hhaavviinngg tthhee

[0028] Similarly, each slice is divided into equally-sized contiguous intervals referred to as units. In some examples each slice is divided into log(k) units. For the sake of simplicity of, in the example of Figure 2, each slice is divided into two units (by the dotted line B). The routing protocol defines a unit leader 22 for each unit. The unit leader is defined dynamically as the router controller having an identifier that is the successor of the identifier at the midpoint of the unit range. If a unit contains an even number of identifiers, such that there is no identifier corresponding to an exact midpoint, a router controller having an identifier closest to the midpoint is defined to be the unit leader.

[0029] Responsive to a new controller joining the peer-to-peer network, the newly-joined controller receives information about the slice leader and the unit leader (e.g. the identifiers of the slice leader and the unit leader) from one of its neighbours, together with other information about the data the neighbour router controller is responsible for, and the routing table of the neighbour router controller.

[0030] Each slice leader 21 performs the following functions:

Receiving event notifications from other router controllers in the peer-to-peer network.

Aggregating received event notifications over a first predetermined time period. The first predetermined time can be based on, e.g., the size of the ring, unit size, technology used by the memory system, congestion, etc. In some examples the first predetermined time period is of the order of a few tens of seconds.

When the first predetermined time period has passed, sending a first aggregate message containing the aggregated event notifications to each other slice leader in the peer-to-peer network. Receiving first aggregate messages from other slice leaders.

Aggregating received first aggregate messages over a second predetermined time period. The second predetermined time period may be empirically determined. In some examples the second predetermined time period is one order of magnitude greater than the first predetermined time period.

When the second predetermined time period has passed, sending a second aggregate message containing the aggregated first aggregate messages to the unit leaders of the units comprised in its slice.

[0031] Each unit leader 22 performs the following functions:

Periodically sending keep-alive messages to its successor.

Periodically sending keep-alive messages to its predecessor.

Receiving second aggregate messages from a slice leader.

Sending a received second aggregate message to its successor and its predecessor together with a keep-alive message.

[0032] Each router controller which is neither a slice leader nor a unit leader performs the following functions:

Periodically sending keep-alive messages to its successor.

Periodically sending keep-alive messages to its predecessor.

Receiving a second aggregate message from one of its successor and its predecessor.

Sending a received second aggregate message to the other one of its successor and its predecessor together with a keep-alive message.

[0033] Thus, in the example, responsive to a router controller 20 detecting the failure of its successor or that it has a new successor (e.g. via the stabilization routine described above), it sends an event notification message 26 to its slice leader 21 . The slice leader 21 collects all event notifications 26 it receives from router controllers 20 in its own slice and aggregates them for a predetermined time period, after which it sends a message 23 containing the aggregated event notifications (a first aggregate message) to each other slice leader. Each slice leader 21 collects all first aggregate messages 23 it receives from other slice leaders 21 and aggregates them for a predetermined time period, after which it sends a message 24 containing the aggregated first aggregate messages (a second aggregate message) to each unit leader 22 in its slice. Each unit leader 22, upon receiving a second aggregate message 24, sends the information 25 contained in the second aggregate message to its successor and its predecessor piggy-backed on a keep-alive message. [0034] Each router controller which is not a slice leader or a unit leader propagates information 25 in a single direction, i.e. if information is received from a predecessor, it is sent on to a successor, and vice versa. Thus, on receiving information 25 from a unit leader 22, each non-leader router controller 20 sends the information 25 on to either their successor (if the unit leader is their predecessor) or their predecessor (if the unit leader is their successor) piggy-backed on a keep-alive message. Router controllers 20 at unit boundaries (i.e. router controllers which do not have a successor, or which do not have a predecessor, within the same unit) do not send information 25 to router controllers outside their unit. In this way, all router controllers in the peer-to-peer network receive all event notifications of all events, but within a unit information always flows from the unit leader 22 to the ends of the unit.

[0035] A consequence of the above-described example routing protocol is that slice leaders 21 have more work to do than the other router controllers. This could create problems in some memory systems, e.g. if they are poorly provisioned system such that a router controller has a massive memory space (i.e. equivalent to a backbone router of an ISP in an IP network). To overcome this problem, in some examples a different routing protocol (hereinafter referred to as a "supercomponent routing protocol") is used. The supercomponent routing protocol defines "supercomponents" as well connected and well provisioned router controllers, e.g. by providing predefined criteria for connectedness and provisionment. A newly-joined router controller is designated as a "supercomponent" during the joining process if that newly-joined router controller meets the predefined criteria.. In such examples a ring of supercomponents is created, parallel to the "ordinary" identifier ring. The slice leader is defined to be the router controller in the supercomponent ring which is the successor of the router controller which is associated with the identifier at the midpoint of the slice range (this midpoint router controller may be in either the supercomponent ring or the ordinary ring). Other aspects of the supercomponent routing protocol are the same as for the example routing protocol described in paragraphs 23-35.

[0036] The example routing protocols therefore enable the fast propagation of notifications around the whole network with reasonable bandwidth consumption, and thus facilitate the maintenance of full routing tables. Example memory systems using one of the example routing protocols can therefore achieve a high query success rate.

[0037] An example of a method for joining a new controller to a peer-to-peer network of a memory system will now be described with reference to Figure 3. In a first block 301 a memory system, e.g. the memory system 1 of Figure 1 , is provided. The memory system comprises a plurality of memory modules, each memory module comprising a non-volatile memory unit and a controller to access information stored in the non-volatile memory unit in response to a request from a processor, to deliver accessed information to the processor, and to communicate with controllers of other memory modules. The controllers are connected so as to form a peer-to-peer network.

[0038] In a second block 302 an additional memory module, comprising an additional controller, is provided in communication with the memory system. The communication may be wired or wireless. In some examples providing an additional memory module in communication with the memory system comprises establishing a communications link between the additional controller and a single controller in the peer-to-peer network. In some examples providing an additional memory module in communication with the memory system comprises establishing a communications link between the additional controller and each of multiple controllers in the peer-to-peer network. The controller(s) in the peer-to-peer network with which a communications link to the additional controller is established may be any controller(s) in the peer-to-peer network. Once a connection has been established, the additional controller accesses a controller in the peer-to-peer network with which it has a communications link. The accessed controller will hereinafter be referred to as the contact controller. At this point the additional controller may be considered to have joined the DHT.

[0039] In some examples, upon joining the DHT, the additional controller discovers the underlying protocol of the memory system, which may be hardwired in firmware and/or hardware. The DHT is maintained by all the controllers in the network in a distributed manner, meaning that any controller can be used as an entry point by additional controllers wishing to join the peer-to-peer network, (i.e. any of the controllers can be the contact controller). In some examples, upon joining the DHT, the additional controller receives information which may comprise, for example, any or all of: a protocol of the memory system, a routing protocol, a routing table, etc.

[0040] When the additional controller has connected to a contact controller and joined the DHT, it receives the memory address range of each controller in the peer-to-peer network, block 303. This enables the additional controller to determine where it should be located in the network topology. For example, a previous controller may have left the ring (in case of failure, for instance), in which case the additional controller may decide to occupy the "hole" left by the previous controller. Or, in another example, the additional controller may decide to extend the current range of addresses of the peer-to-peer network beyond its current upper bound.

[0041] Then, in block 304, using the received address information the additional controller connects to a first controller, where the first controller is another controller in the peer-to-peer network (i.e. other than the contact controller). The memory address ranges received by the additional controller in block 303 are sent in messages which also identify the position of the sender (i.e. the controller which has sent a given message) in the network topology. Using this position information the additional controller determines, in block 305, whether a direct connection exists between it and the first controller. Then, in block 306, if it is determined that a direct connection exists with the first controller, the additional controller determines the role of the first controller. Specifically, it determines whether the first controller is router controller, a switch controller, or a plain memory controller. This determination is performed on the basis of information advertised by the first controller. In some examples the additional controller performs blocks 304, 305 and 306 in respect of each of multiple controllers.

[0042] In a final block 307, a role is selected for the additional controller from the roles of router, switch and plain memory. The selection is based on whether a direct connection exists between the additional controller and the first controller, and if a direct connection exists, additionally on the role of the first controller, wherein the selected role is selected from the group: router, switch, plain memory. The rules used in the selection are set out by a protocol of the memory system and are stored by each controller in the peer-to-peer network. In an example, an additional controller that is directly connected to a switch controller or a router controller is selected to be a plain memory controller, whereas an additional controller that is not directly linked to router controller or a switch controller is selected to be a router controller. In the example an additional controller will be selected to be a switch if it is connected to a router controller and more than a predetermined minimum number of plain memory controllers. The predetermined minimum number may be based on, for example, any or all of: the overall memory, the memory system technology, and the memory system protocol. For instance, a memory system comprising more than 10 memory modules, with more than 1 PB of storage, will comprise several switches.

[0043] If an additional controller is selected to be a plain memory controller, in accordance with the rules of the memory system protocol it prunes non-direct links to other controllers, and it negotiates its address range with the closest router controller or switch controller.

[0044] In some examples the method additionally involves the additional controller contacting the first controller and receiving a second range of memory addresses from the first controller. The second range of memory addresses are associated with a second controller of the memory system, which is directly connected to the first controller. The additional controller then determines whether a direct connection exists between it and the second controller; and if a direct connection exists, determines the role of the second controller. In such examples the role of the additional controller is selected based additionally on whether a direct connection exists between the additional controller and the second controller, and if a direct connection exists, additionally on the role of the second controller.

[0045] A specific example of the method of Figure 3 could be performed as follows. Once an additional controller of an additional memory module has connected to a contact controller, the contact controller sends out a handful of controller memory ranges addresses. The additional controller consequently tries to connect to the controllers associated with the received memory range address, and in response these controllers also give the additional controller the addresses of controllers connected to them. This continues until a predetermined maximum number of hops has been reached, wherein the maximum is set down by the memory system protocol.

[0046] Then, the additional controller uses the memory system protocol to determine whether it has a direct connection to any of these controllers for which it has received memory address ranges. If there is, it uses information published by the directly connected controllers to determine the roles of the directly connected controllers (i.e. router, switch or plain memory), and a role for the additional controller is selected accordingly. Once a role has been associated with the additional controller, configuration processes of the memory system are performed (e.g. if a plain memory controller is connected to a switch controller, the switch controller runs a protocol similar to a spanning tree in an IP network to auto- configure the switch).

[0047] In principle, knowing a single controller address (e.g. of the contact controller) is sufficient for a joining additional controller to obtain addresses for all of the other controllers in the peer-to-peer network, since the contact controller will share the rest of the addresses with the additional controller. This reduces the bandwidth used by the DHT.

[0048] An example of a method for accessing information stored in a memory system will now be described with reference to Figure 4. In a first block 401 a plurality of memory modules is provided. Each memory module comprises a non-volatile memory unit and a controller, and the controllers are connected in communication so as to form a peer-to-peer network. Each memory module is associated with a memory address range, the memory address range of a given memory module being unique among the set of its neighbouring memory modules in the peer-to-peer network. In some examples the memory modules are comprised in a memory system, e.g. the memory system 1 of Figure 1.

[0049] In a second block 402 a first controller of a first memory module receives a request from a processor for information stored in one of the other memory modules. In some examples the requested information is stored in just one of the memory modules. In some examples the requested information is stored in multiple memory modules. [0050] Then, in block 403, the first controller sends the request to another controller, or multiple other controllers, in the peer-to-peer network. Which controller(s) the request is sent to is determined by the first controller in accordance with a routing protocol of the peer-to- peer network. In some examples the controller sends the request to its predecessor and its successor in an identifier ring. In some examples, the controller sends the request to its predecessor or to its successor. Block 403 will be performed by each controller which receives the request, until the request has reached all of the controllers in the peer-to-peer network. Each controller, upon receiving the request, determines whether any of the requested information is stored by its memory module. A controller of a memory module which stores at least part of the requested information in its NVM unit becomes a second controller for the purposes of the method.

[0051] In block 404, a second controller of a second memory module receives the request from another controller in the peer-to-peer network. At least part of the requested information is stored in the NVM unit of the second memory module. Then, in block 405, the second controller determines that at least part of the requested information is stored in the non-volatile memory unit of the second memory module, as described above. In response to this determination, the second controller retrieves the requested information from the NVM unit of the second memory module and sends the requested information to another controller, or multiple other controllers, in the peer-to-peer network according to the routing protocol, block 406. Each controller which receives the requested information will forward it to at least one other controller, according to the routing protocol, until the requested information has reached every controller in the peer-to-peer network.

[0052] In block 407 the first controller receives the requested information from another controller in the peer-to-peer network (i.e. a second controller) and in block 408 the first controller provides the requested information to the processor. The first controller may receive the requested information from multiple other controllers in the peer-to-peer network. In such cases, in some examples the first controller provides the requested information to the processor response to the first receipt of the requested information, and does not provide the requested information to the processor responsive to subsequent receipts of the same requested information. The first controller may receive a first part of the requested information from a first other controller and a second part of the requested information from a second other controller, and/or may receive a first part of the requested information at a different time from a second part of the requested information. In such cases, in some examples the first controller waits until all parts of the requested information has been received. Response to all parts of the requested information having been received by the first controller, the first controller provides the requested information to the processor. [0053] In the case where a request is received by a memory module for information stored in that module, it is not necessary to perform the above described method. In such cases the memory module receiving the request will simply retrieve the requested information from its NVM unit and deliver it to the requesting processor.

[0054] If a memory access request from a processor to a given memory module controller fails because that memory module is no longer in the system, the processor can retry fetching the data by sending the memory access request to the successor of the given memory module.

[0055] Each of the example memory systems described above can therefore be seen as a routing overlay built on top of a dynamic set of memory components, such that the system has routing, switching and data access capabilities.

[0056] The present disclosure is described with reference to flow charts and/or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. It shall be understood that each flow and/or block in the flow charts and/or block diagrams, as well as combinations of the flows and/or diagrams in the flow charts and/or block diagrams can be realized by machine readable instructions.

[0057] The machine readable instructions may, for example, be executed by a general purpose computer which forms a special purpose computer upon having the machine readable instructions stored and executed thereon, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine readable instructions. Thus functional modules of the apparatus and devices may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate array etc. The methods and functional modules may all be performed by a single processor or divided amongst several processors.

[0058] Such machine readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. [0059] Such machine readable instructions may also be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operation steps to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide a step for realizing functions specified by flow(s) in the flow charts and/or block(s) in the block diagrams.

[0060] While the method, apparatus and related aspects have been described with reference to certain examples, various modifications, changes, omissions, and substitutions can be made without departing from the spirit of the present disclosure. It is intended, therefore, that the method, apparatus and related aspects be limited only by the scope of the following claims and their equivalents. It should be noted that the above-mentioned examples illustrate rather than limit what is described herein, and that those skilled in the art will be able to design many alternative implementations without departing from the scope of the appended claims.

[0061] The word "comprising" does not exclude the presence of elements other than those listed in a claim, "a" or "an" does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims.

[0062] The features of any dependent claim may be combined with the features of any of the independent claims or other dependent claims.

Claims

1. A memory system comprising a plurality of memory modules, wherein each memory module comprises:

a non-volatile memory unit; and

a controller to access information stored in the non-volatile memory unit in response to a request from a processor, to deliver accessed information to the processor, and to communicate with controllers of other memory modules;

wherein the controllers are connected so as to form a peer-to-peer network of memory modules, and wherein each given memory module is associated with a memory address range which is non-overlapping with a memory address range associated with any memory module to which the given memory module is directly connected.

2. A memory system in accordance with claim 1 , wherein each memory module is associated with a unique memory address range.

3. A memory system in accordance with claim 2, wherein the memory address range associated with a given memory module is determined during of joining of the controller of that memory module to the peer-to-peer network.

4. A memory system in accordance with claim 1 , wherein each controller is to advertise the memory address range of its memory module to the other controllers in the peer-to-peer network.

5. A memory system in accordance with claim 1 , wherein each controller is associated with a role, the role being one of: router, switch, plain memory.

6. A memory system in accordance with claim 5, wherein each router controller is to store a routing table containing an identifier for every other router controller in the peer-to-peer network and to route memory access requests according to the stored routing table; and each switch controller is to forward memory access requests to every other controller in the peer-to-peer network.

7. A memory system in accordance with claim 1 , wherein each router controller is associated with a unique identifier, and wherein the peer-to-peer network is to store the identifiers in a distributed hash table, DHT.

8. A memory system in accordance with claim 7, wherein the identifiers are uniformly distributed in an identifier space.

9. A memory system in accordance with claim 7, wherein the identifiers are ordered in an identifier ring, such that each router controller has a predecessor and a successor in the identifier ring.

10. A memory system in accordance with claim 9, wherein:

the identifier ring comprises a plurality of equally-sized contiguous slices, each containing a slice range of identifiers, and wherein for each slice a slice leader is defined as: a router controller associated with an identifier that is the successor of the identifier at the midpoint of the slice range; or, if there is no identifier at the midpoint, a router controller having an identifier closest to the midpoint; and

each slice comprises a plurality of equally-sized contiguous units, each containing a unit range of identifiers, and wherein for each unit a unit leader is defined as: a router controller associated with an identifier that is the successor of the identifier at the midpoint of the unit range; or, if there is no identifier at the midpoint, a router controller having an identifier closest to the midpoint.

1 1. A memory system in accordance with claim 10, wherein each slice leader is to:

receive event notifications from other router controllers in the peer-to-peer network; aggregate received event notifications over a first predetermined time period;

when the first predetermined time period has passed, send a first aggregate message containing the aggregated event notifications to each other slice leader in the peer-to-peer network;

receive first aggregate messages from other slice leaders;

aggregate received first aggregate messages over a second predetermined time period;

when the second predetermined time period has passed, send a second aggregate message containing the aggregated first aggregate messages to the unit leaders of the units comprised in its slice.

12. A memory system in accordance with claim 1 1 , wherein each unit leader is to: periodically send keep-alive messages to its successor;

periodically send keep-alive messages to its predecessor;

receive second aggregate messages from a slice leader; and

send a received second aggregate message to its successor and its predecessor together with a keep-alive message.

13. A memory system in accordance with claim 12, wherein each router controller which is not a unit leader or a slice leader is to:

periodically send keep-alive messages to its successor;

periodically send keep-alive messages to its predecessor;

receive a second aggregate message from one of its successor and its predecessor; and

send a received second aggregate message to the other one of its successor and its predecessor together with a keep-alive message.

14. A memory system in accordance with claim 1 , wherein the memory system comprises non-volatile memory units which differ from each other in respect of at least one of: capacity; manufacturer; storage technology; memory protocol, memory system technology.

15. A method, comprising:

providing a plurality of memory modules, each memory module comprising a nonvolatile memory unit and a controller; wherein the controllers are connected in

communication so as to form a peer-to-peer network, and wherein each memory module is associated with a memory address range, the memory address range of a given memory module being unique among the set of its neighbouring memory modules in the peer-to-peer network;

a first controller of a first memory module receiving a request from a processor for information stored in at least one of the other memory modules;

the first controller sending the request to at least one other controller in the peer-to- peer network according to a routing protocol of the peer-to-peer network;

a second controller of a second memory module receiving the request from another controller in the peer-to-peer network, wherein at least part of the requested information is stored in the non-volatile memory unit of the second memory module;

the second controller determining that at least part of the requested information is stored in the non-volatile memory unit of the second memory module; the second controller retrieving the requested information from the non-volatile memory unit of the second memory module and sending the requested information to at least one other controller in the peer-to-peer network according to the routing protocol;

the first controller receiving the requested information from another controller in the peer-to-peer network; and

the first controller providing the requested information to the processor.