EP2979191B1 - Koordination der replikation von in einem nichtflüchtigen speichersystem gespeicherten daten - Google Patents

Koordination der replikation von in einem nichtflüchtigen speichersystem gespeicherten daten Download PDF

Info

Publication number
EP2979191B1
EP2979191B1 EP13879729.5A EP13879729A EP2979191B1 EP 2979191 B1 EP2979191 B1 EP 2979191B1 EP 13879729 A EP13879729 A EP 13879729A EP 2979191 B1 EP2979191 B1 EP 2979191B1
Authority
EP
European Patent Office
Prior art keywords
volatile memory
stream
storage
writes
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP13879729.5A
Other languages
English (en)
French (fr)
Other versions
EP2979191A1 (de
EP2979191A4 (de
Inventor
Douglas L. Voigt
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hewlett Packard Enterprise Development LP
Original Assignee
Hewlett Packard Enterprise Development LP
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett Packard Enterprise Development LP filed Critical Hewlett Packard Enterprise Development LP
Publication of EP2979191A1 publication Critical patent/EP2979191A1/de
Publication of EP2979191A4 publication Critical patent/EP2979191A4/de
Application granted granted Critical
Publication of EP2979191B1 publication Critical patent/EP2979191B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2064—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring while ensuring consistency
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2082—Data synchronisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023—Free address space management
    • G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
    • G06F12/0246—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory in block erasable memory, e.g. flash memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0688—Non-volatile semiconductor memory arrays
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/30003—Arrangements for executing specific machine instructions
    • G06F9/30076—Arrangements for executing specific machine instructions to perform miscellaneous control operations, e.g. NOP
    • G06F9/30087—Synchronisation or serialisation instructions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1415—Saving, restoring, recovering or retrying at system level
    • G06F11/1441—Resetting or repowering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/10—Providing a specific technical effect
    • G06F2212/1016—Performance improvement
    • G06F2212/1024—Latency reduction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/72—Details relating to flash memory management
    • G06F2212/7202—Allocation control and policies
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/72—Details relating to flash memory management
    • G06F2212/7208—Multiple device management, e.g. distributing data over multiple flash devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/72—Details relating to flash memory management
    • G06F2212/7209—Validity control, e.g. using flags, time stamps or sequence numbers
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • a computer system traditionally has contained both volatile and non-volatile storage devices.
  • volatile memory devices such as dynamic random access memory (DRAM) devices
  • DRAM dynamic random access memory
  • US 2007/011213 A1 discloses systems and methods of asynchronous data replication.
  • a primary host asynchronously transmits write and sync commands where each write and sync command includes a data set ID and a sequence number.
  • a secondary host coupled to the primary host receives the write and sync commands out of order, tracks receipt of write and sync commands, and after receipt of the write and sync commands of the data set writes blocks associated with the write commands to data storage coupled to the secondary host.
  • the NVM-based computer system includes a primary NVM storage, which serves as the main (i.e., the "working") memory for the computer system; and a secondary NVM storage, which stores replicated data (i.e., data that is redundant to the data stored in the primary NVM storage).
  • the computer system generates a first stream of writes to the primary NVM storage to store data in the primary NVM storage, and this first stream of writes includes synchronization commands to create corresponding synchronization points.
  • the computer system is constructed to generate a second stream of writes to the secondary NVM storage and coordinate the first and second streams in a manner that allows recoverability from single points of failure, while minimizing memory access latency times.
  • Fig. 1 depicts a computer system 100 according to an example implementation.
  • the computer system 100 includes an application server 110, which is an actual physical machine that is made up of actual hardware and actual machine executable instructions, or "software.”
  • the application server 110 includes one or multiple central processing units (CPUs) 114 (multiple CPUs 114, being depicted in Fig. 1 , as an example), which may each contain one or multiple processing cores.
  • the application server 110 also includes an NVM-based storage subsystem 120, which contains NVM storage 124-1 that serves as the main, or primary, working memory for the application server 110.
  • the CPUs 114 execute machine executable instructions to form one or more software-based components 135 for the application server 110, such as an example application 136, a file/object store 138 created and manipulated by the application 136, an NVM manager 140 and booting/basic input operating system (BIOS) firmware 142.
  • software-based components 135 for the application server 110, such as an example application 136, a file/object store 138 created and manipulated by the application 136, an NVM manager 140 and booting/basic input operating system (BIOS) firmware 142.
  • BIOS basic input operating system
  • the application 136 manipulates various files and objects, which are stored in the primary NVM storage 124-1.
  • a failure or abrupt power loss may result in the loss of data that has been manipulated by the application 136 but not yet stored, or committed, to the primary NVM storage 124-1. Therefore, during its course of operation, the application 136 generates synchronization commands for purposes of assuring sufficient data consistency to guarantee recoverability in the event of failure or abrupt power loss.
  • a given synchronization command causes the storage subsystem 120 to acknowledge that a certain number of writes that are identified by the synchronization command have been committed to the primary NVM storage 124-1.
  • a "write" refers to the communication of data, the appropriate write command and the address to the storage subsystem 120 for the data to be stored in the primary NVM storage 124-1.
  • the synchronization points created by the synchronization commands are used, in accordance with example implementations disclosed herein, for purposes of assuring that redundant copies of the data are stored in a secondary NVM storage, as further disclosed herein.
  • the computer system 100 includes a storage server 146, which for this example contains an additional NVM-based storage subsystem 120-2 that contains two additional NVM storages 124-2 and 124-3.
  • the NVM storage 124-1 of the application server 110 may be considered the "primary" NVM storage
  • the NVM storage 124-2 may be considered the "secondary” NVM storage
  • the NVM storage 124-3 may be considered a third NVM storage, in accordance with an example implementation.
  • the storage server 146 is an actual physical machine that is made up of actual hardware and software.
  • the hardware of the storage server 146 includes such components as one or multiple CPUs 147, and the NVM storages 124-2 and 124-3.
  • the CPU(s) 147 may execute machine executable instructions to form software components 160, such as a data service component 162, file(s)/object(s) 164, an NVM manager 166 and boot up/BIOS firmware 168.
  • writes to a primary copy of the data stored in the primary NVM storage 124-1 may be streamed to the secondary NVM system 124-2 synchronously, or marked for replication.
  • the storage server 146 receives, or finishes receiving, all of the writes to be replicated along with the atomicity information related to the synchronization operation.
  • Execution of the synchronization command completes after both copies are updated. Although both copies may be updated by the end of the synchronization command execution, the copies are not necessarily committed together. This means that, in accordance with some implementations, a power or hardware failure may force one copy or the other to be used in its entirety for recovery.
  • Techniques are disclosed herein to allow recovery from a combination of the primary 124-1 and secondary 124-2 NVM storages, such as consistency group and logging techniques, as further disclosed herein.
  • the secondary NVM storage 124-2 may not be the only source for replicated data.
  • the storage server 146 creates additional redundancy in the background.
  • the storage server 146 may perform a background data service, such as redundant array of inexpensive disk (RAID)-based operations, to copy the data stored in the secondary NVM storage 124-2 to the NVM storage 124-3.
  • This background process may be coordinated with unmap operations in such a way that the primary copy is not unmapped and evicted from the primary NVM storage 124-1 until the secondary copy is redundant.
  • techniques and systems are disclosed herein to create three copies of data (a primary copy and two redundant copies).
  • the rate of secondary replication may be controlled using a recovery point objective to describe the maximum amount of work that may be lost in the event of a failure after an unmap command.
  • implementations are disclosed herein, which allow various data services, such as remote replication, point in time copying and data deduplication, to lag slightly behind the application's memory accesses while still operating on consistency points that are relevant to the applications.
  • NVM storage refers to memory that is, or can be, read or written on a per-byte or per-processor-cache line basis.
  • a synchronize operation may be used to assure sufficient data consistency to assure recoverability in the event of a failure or abrupt power loss.
  • the synchronization points are further used to assure that a redundant copy of the data is also in place.
  • Fig. 1 depicts two servers, in further implementations, one or multiple additional application servers and/or one or multiple additional storage servers may be employed.
  • the servers may be blade servers in a common chassis.
  • any component that is shared between the servers may be redundantly provided by the chassis or other components within it.
  • the hardware and software components depicted in Fig. 1 may be disposed on a single physical server.
  • the applications such as application 136
  • the data services such as example data service 162
  • Applications may consume storage related services within the application server 110, such as the file systems, object storage or databases, are collectively illustrated by reference numeral 138.
  • These storage-related services may, in turn, use the NVM management services, provided by the NVM manager 140 of the application server 110, and/or drivers to control and coordinate access to the primary NVM storage 124-1, although reads and writes may take the form of load and store CPU instructions or other instructions with operands that call for memory access.
  • the boot/BIOS components 142 are constructed to accommodate the NVM memory storage 124-1.
  • the data service component 162 may access file, object, database and NVM management services that are provided by an NVM manager 166.
  • the vertical arrow of Fig. 1 traversing the application and storage server software components of both servers 110 and 146 indicates a flow of NVM-related commands, which originate at multiple levels of the stack.
  • the data stored in the NVM storages 124-1, 124-2 and 124-3 may be encrypted data, and as such, the computer system 100 may contain the appropriate encryption and decryption engines, which allows protection in the presence of removable non-volatile private data.
  • the data stored with the NVM storages 124-1, 124-2 and 124-3, for the examples disclosed herein, are organized in volumes 130.
  • the NVM storage 124-1 of the application server 110 includes two example volumes 130-1 and 130-2.
  • the NVM memory storage 124 may contain more or less than two volumes, depending on the particular implementation and application.
  • other data containment schemes may also apply, in accordance with further implementations.
  • the application server 110 contains L volumes 130 with various numbers of objects 134 contained therein.
  • the two volumes 130 contain M and N objects, respectively.
  • the NVM storages 124-2 and 124-3 of each storage server contains volumes and objects, which correspond to the volumes and objects of the NVM storage 124-1, in accordance with example implementations.
  • the notation “NVM” refers to non-volatile memory storage in the primary NVM storage 124-1
  • the notation “NVM'” refers to the NVM storage in the secondary NVM storage 124-2
  • the notation “NVM”” refers to the NVM storage in the NVM storage 124-3.
  • the objects are represented with corresponding notations.
  • volume 1 refers to volume one of the NVM storage 124-1
  • Vol 1' represents the corresponding volume one stored in the secondary NVM' storage
  • Vol 1" refers to the corresponding volume stored in the NVM” storage.
  • either the application server 110 or the storage server 146 may have actual NVM allocated for only a subset of the indicated objects at a given point in time.
  • the NVM' and NVM" storages may be in separate fault domains, meaning that the storages may share no single points of failure.
  • each of the NVM, NVM' and NVM" storages may have internal redundancy for purposes of recovering failed subsets of the NVM storage.
  • These internal redundancy features may include error correction code (ECC) and erasure coding techniques, such as RAID, which provide additional protection against NVM media failure but may not tolerate all single points of failure within their respective servers.
  • the CPUs and NVM storage systems interact locally using load, store and other instructions, whose operands reference memory.
  • the majority of NVM reads may be satisfied locally within the application server 110.
  • the writes are, in general, continuously streamed from the application server 110 to the storage server 146.
  • the synchronization operations which coordinate NVM storage to assure accurate recovery from failure may flow either through the software stack or directly through the NVM hardware, depending on the particular implementation. In either case, synchronization operations cross the application/server boundaries.
  • Some read operations on the application server 110 may access data that does not currently reside in the application server's NVM storage. In this case, the data may be retrieved, or read, from the NVM' storage on the storage server 146.
  • These reads use access redirection in hardware and/or virtual address space management in software, similar to page fault handling techniques, as can be appreciated by the skilled artisan.
  • the computer system 100 includes a storage manager 180.
  • the storage manager 180 coordinates the configuration of the NVM aspects of the application 110 and storage 146 servers. These responsibilities include the creation and deletion of corresponding volumes or other containers on both the application 110 and storage 146 servers.
  • the storage manager 180 may be disposed on a separate server (e.g., on a separate physical machine, for example), other than the machines containing the application 110 and storage servers 146; or may be part of the application 110 or storage 146 server.
  • the storage manager 180 may be part of the storage solution or the operating system (OS) managing the storage server 146.
  • the storage manager 180 may take the form of one or multiple plug-ins or widgets in the context of a broader management infrastructure, such as a self-service portal or a converged infrastructure suite.
  • the following timeline illustrates the operation of the computer system 100 in accordance with an example implementation.
  • the computer system 110 may be operated in accordance with other timelines that support the coordination processes, as can be appreciated by the skilled artisan.
  • the storage manager 180 creates an NVM volume, to which the application server 110 and the storage server 146 are made aware.
  • the operating system of the application server 110 creates a file system that uses the volume or designates the space in the volume to be available to the application 136 via an NVMalloc operation, for example, which involves both the application and storage server software.
  • the application 136 creates and opens a file or object (example objects "OBJ1,” “OBJ1M,” and so forth being depicted in Fig. 1 ) or is given access to NVM space through an NVMalloc service.
  • the application 136 or file/object software component 138 allocates NVM space for its use and maps it into the CPU's virtual memory space.
  • the allocation and mapping information are communicated to the storage server 146, which creates corresponding "objects" (example objects "OBJ1',” “OBJM',” and so forth being depicted in Fig. 1 ) herein.
  • the allocation and mapping may be combined as one operation.
  • mapping may take place during the creation and opening of a file or object or concurrent with the NVMalloc service.
  • the application 136 accesses objects using CPU instructions with operands that reference the NVM storage. When memory locations are written, the CPU 114 initiates a corresponding communication with the storage server NVM manager 166. This communication may or may not include write data, depending on the particular implementation. The application server 136 may or may not wait for a response from the storage server 146 before viewing the write as complete.
  • the read request is satisfied, if possible, from the application server's NVM space. If not in the NVM space, however, the data may be read from the NVM' space on the storage server 146.
  • the application 136 may occasionally generate a synchronization command for purposes of expressing a recovery point.
  • the synchronization command may be generated via an explicit call or it may be implicit in some use of the language in which the application 136 is written, depending on the particular implementation.
  • the synchronization operation may traverse either a software path or a hardware path to the NVM storages 124-1 and 124-2.
  • the synchronization implementations may include flush instructions, which are executed by the application CPU 114 and/or synchronization commands that are communicated to the NVM systems 124-1 and 124-2 via load or store instructions.
  • the application 136 or the operating system may issue an unmap command, which is communicated to both the application 110 and storage 146 servers. In this manner, when the unmap command completes, the application 136 no longer has direct access to the data.
  • An unmapping may correspond with such events as the closing of a file, the halting of the application 136 or the operating system's need to free up memory address space for other purposes. It is noted that the data may be mapped again later.
  • the application 136 or the operating system may evict the object from the NVM storage 124-1. If the object is still mapped and is subsequently read from or written to, the data is read from the NVM' storage into newly allocated physical space in the NVM storage 124-1, similar to paging systems. If the object is not mapped, then new space may be allocated in the NVM storage 124-1, and the mapping re-establishes the same data previously associated with the object and still stored on the storage server 146.
  • a volume may no longer need to be retained, and for this scenario, the storage server 146 may take actions to delete the volume from both the application server 110 and the storage server 146.
  • NVM' and NVM" storages may reside on different tiers of NVM or rotating media.
  • other background processes may be performed, other than processes that maintain redundancy, in accordance with further implementations.
  • the same timelines and trigger points disclosed herein may be used to cause background processing for services, such as index creation, point in time copying or data deduplication.
  • operational flow begins at time T 0 with the application 136 executing a stream 310 of byte extent or cache line writes.
  • the writes are associated with object Obj1.
  • the NVM system 124-1 initiates write streaming 314 to the corresponding object Obj1' shortly after writing the Obj1 object.
  • the application 136 issues a synchronization command 311.
  • the application 136 continues generating writes during the synchronization operation, although the application 136 is aware that the synchronization operation has not completed.
  • the application 136 avoids corrupting the data being synchronized by waiting for the synchronization operation to complete if any concurrent writes conflict with each other.
  • the storage server 146 does not respond to the synchronization command 311 until the server 146 has received all of the writes up to and including any writes that are referenced by the synchronization command (writes W1 to W5 for this example). Although not depicted in Fig. 3 , it is possible that a write that originated after the synchronization command may arrive before the synchronization operation completes. As with the application server's image, it is the application server's responsibility to avoid this scenario if it matters.
  • the storage server 146 responds to, or acknowledges 320, the synchronization command.
  • the storage server 146 responds to the originator of the synchronization command (from the point of view of the server 146), which may be, for example, hardware associated with Obj1 object or the data service 162 for the case in which the synchronization command was delivered via software.
  • the final acknowledgment step is illustrated in Fig. 3 by the "Sync5" arrow 325 pointing from Obj1 object to "App.”
  • the NVM storage 124-2 notifies the data service 162 that the synchronization command has been completed. This triggers a background process, which creates redundancy for the recently synchronized data within the storage server 146, as illustrated by "RAID 1-5" at reference numeral 330 in Fig. 3 .
  • the "1-5" designation indicates that the storage server 146 has a record of the writes that are to be made internally redundant and also has associated them with the synchronization command.
  • This background replication includes not only the writes indicated by the synchronization command but all preceding writes as well. Subsequent writes to the same addresses in the Obj1 and Obj1' objects may cause temporary inconsistency in the RAID redundancy. This may be tolerated because the RAID redundancy may be ignored in accordance with example implementations until the corresponding objects are unmapped, at which point RAID consistency is assured.
  • the record of writes to be made redundant as a result of a given synchronization command may be retained in any number of ways, with each having a different metadata size and synchronization overhead characteristics. For example, in accordance with some implementations, a log of specific byte ranges written may be retained. In further implementations, a log of larger regions of storage containing the writes may be used. In accordance with further implementations, a bit map of fixed size regions may be maintained. In general, the larger the region size, the smaller the tracking structure and the more data that was not recently written is synchronized. In the case, the tracking structure is not a non-volatile storage, power failure may force resynchronization of all data just in case it was written.
  • a goal of the background replication process may be to create at least three effective copies of the data before the primary copy (such as "Obj1" object in the example in Figs. 2-3 ) is evicted or unmapped from the application server memory.
  • the time used to free space in the application server memory may be greatly reduced, as compared with scenarios where the entire object is replicated before eviction. This is illustrated near time T 4 in Fig. 3 , when the application generates (reference numeral 350) an unmap command.
  • the data service 162 may expedite completion of the remaining synchronization activity related to the Obj1".
  • the data service 162 responds to the unmap command, and the Obj1 object may be safely removed from the application server 136, due to a fault tolerant copy being stored on the storage server 146. After this point, the Obj1" object may be used to recover the Obj1' object.
  • the RAID algorithm may span more than two devices, which here, are multiple NVMs.
  • the "lazy" RAID process spans additional NVMs as needed.
  • a single RAID stripe may not include more than one object that can be unmapped separately. Otherwise, parity information for other unmapped objects in the same stripe may become inconsistent due to ongoing activity in objects that are still mapped.
  • One way to minimize this effect is to disable the lazy RAID on partially mapped stripes by expediting RAID so that the RAID completes during the synchronization operation.
  • the exposure gap may be completely closed if a log is used, as further disclosed below.
  • a technique 400 includes generating (block 402) a stream of writes to a secondary non-volatile memory system in response to streams of writes to a primary non-volatile memory system.
  • the technique 400 includes embedding (block 404) one or more commands in the second stream to create one or more synchronization points for data storage in the secondary non-volatile memory system in response to the command(s) in the first stream to create synchronization point(s) for data storage in the primary non-volatile memory system.
  • Fig. 5 illustrates the combined storage subsystem 200 for purposes of restoring a consistency group.
  • the storage server 146 may not have a record of the bytes written within the application server 135 between the completion of the last synchronization operation and the point in time of failure.
  • the application's recovery algorithm includes rolling back memory state to the most recently completed synchronization command, the entire NVM image that is relevant to the application 136 is restored.
  • the grouping of all data relevant to the application 136 for this purpose is called a "consistency group," such as consistency group 500 of Fig. 5 .
  • the consistency group 500 may span multiple volumes and may include arbitrary objects, partially or in their entirety, within multiple volumes.
  • the consistency group 500 may also include all of the corresponding replicas within the storage server 146.
  • the composition of each consistency group 500 may be part of the non-volatile configuration state of the storage server 146.
  • a replacement application server inherits the consistency group 500, establishes a mapping that corresponds to that used by the old server, recovers the consistency group 500 to a consistent state, and begins using the data in the consistency group 500.
  • This process may be orchestrated by the storage manager of an OS failover component in conjunction with the storage manager, depending on the particular implementation.
  • the storage manager may send a background restore command to the data service, for example.
  • reads may be demand paged from the consistency group replicas in the storage server 146.
  • a background restoration process may be used to place the desired data, if known, back in the application server NVM so that the data may be present without demand paging when needed.
  • the combined storage subsystem 200 may include an NVM-based log 600.
  • the log 600 receives writes at the same time as writes to the NVM storage 124-2. Because the log 600 is in NVM on the storage server 146, there is a replica of every write before completion of the initial write to the NVM storage 124-2. The use of the log 600 minimizes the chance that a short term lapse in the operation of NVM storage 124-2 or assures that congestion that is related to NVM storage 124-2 does not interfere with the application operation.
  • the log 600 is illustrated as being a separate component from the NVM' storage, as it is intended to be smaller, more reliable and generally lower power than the NVM' storage. This is to maximize the probability that the log will be accurate and up to date in any failure or power lost scenario other than those that affect the component or the entire storage server.
  • the log 600 may be a first in first out (FIFO) buffer, that has parity protection so that the log 600 may operate at the line rate of the application server interface.
  • the log 600 may also record the address and data of every write.
  • the log 600 may collapse re-writes of a given address from the application server 135, as long as there was no intervening synchronization operation. This is illustrated in Fig. 7 of an example workflow 700. In this manner, write number six (W6) occurs twice (as illustrated at reference numerals 704 and 708) with different values but gets consolidated in the log 600.
  • the size of the log 600 may be calculated by setting the goals for the amount of temporary delay in the NVM' storage, which may be tolerated and the maximum transfer rate from the storage server 146.
  • maximum transfer rates for multiple lane photonic interconnects might be, as an example, 400 gigabits per second (Gbits/s).
  • Gbits/s gigabits per second
  • Kb kilobytes
  • the average transfer rates in and out of the log 600 must relatively be matched in such a way that the 1 ⁇ s lag is not exceeded. The worst case lag is also affected by potential stalls in the outflow of data from the FIFO.
  • the log 600 may be used to track data that was accessed between synchronization operations after the operations complete. For example, a synchronization command may arrive every 10 ⁇ s and take about 100 ⁇ s on average to complete. This means that the storage server 146 processes ten synchronization operations at a time. Furthermore, the storage server 146 contains enough data to cover 111 ⁇ s of write data, which is a log size of 5.55 megabits (Mb). To operate in this mode, the log 600 retains the data associated with each synchronization command until that synchronization operation completes. The data may be used by the synchronization operation to ensure that everything related to the synchronization operation is accounted for under a maximum number of failure scenarios.
  • Mb 5.55 megabits
  • the log may provide a complete accounting of log data and may be combined with expedited RAID to accommodate partially mapped RAID stripes. This allows correction of the parity in such stripes after a power loss.
  • parity consistency management techniques may be employed. For example, two versions of parity may be retained in the NVM storage prior to the writing of new data. At the point when the new version of parity is completely generated, that completion becomes the commit point of the stripe update. From that point on, the new data in the log 600, which was not yet written to NVM' storage is assumed to be correct. If the new version of the parity was not completed, then the old version of parity and the old data are used. Coordinated data and parity commitment embedded in a formal transaction and timestamped versions of data and parity may also be used.
  • the log 600 may be used for purposes of tuning shadowing, i.e., tuning a timing to allow the shadowing of updates in the NVM storage by one or more synchronization operations in the NVM' storage. This allows atomicity of synchronization operations in NVM' storage with respect to visibility across servers when NVM' storage is shared.
  • the shadowing is illustrated in connection with an example workflow 800 of Fig. 8 .

Claims (15)

  1. Verfahren, Folgendes umfassend:
    als Reaktion auf einen ersten Strom von Schreibvorgängen in ein erstes nicht flüchtiges Speichersystem (124-1), Erzeugen (402) eines zweiten Stroms von Schreibvorgängen für ein zweites nicht flüchtiges Speichersystem (124-2); und
    Koordinieren des Replizierens von im ersten nicht flüchtigen Speichersystem (124-1) gespeicherten Daten, umfassend das Einbetten (404) wenigstens eines Befehls in den zweiten Strom von Schreibvorgängen zum Erzeugen eines Synchronisierungspunkts für Datenspeicherung im zweiten nicht flüchtigen Speichersystem (124-2) als Reaktion auf wenigstens einen entsprechenden Befehl im ersten Strom zum Erzeugen eines Synchronisierungspunkts für Datenspeicherung im ersten nicht flüchtigen Speichersystem (124-1).
  2. Verfahren nach Anspruch 1, ferner Folgendes umfassend:
    Durchführen eines Datendienstes auf einem dritten nicht flüchtigen Speichersystem (124-3) als Reaktion auf den zweiten Strom von Schreibvorgängen.
  3. Verfahren nach Anspruch 2, wobei das Durchführen des Datendienstes das Durchführen von Indizierung, Deduplizierung, Komprimierung, Fernreplikation, Point-in-Time-Kopieren und/oder das Durchführen von Schreibvorgängen basierend auf redundanter Anordnung unabhängiger Festplatten (redundant array inexpensive disks - RAID) in das dritte nicht flüchtige Speichersystem (124-3) umfasst.
  4. Verfahren nach Anspruch 1, ferner Folgendes umfassend:
    Protokollieren des zweiten Stroms von Schreibvorgängen in das zweite nicht flüchtige Speichersystem (124-2), wobei das Protokollieren das Speichern von Daten umfasst, die auf den zweiten Strom in einer Warteschlange des zweiten nicht flüchtigen Speichersystems (124-2) hinweisen; und
    Abrufen der Daten aus der Warteschlange und Durchführen der Schreibvorgänge in dem zweiten nicht flüchtigen Speichersystem (124-2).
  5. Verfahren nach Anspruch 4, wobei das Protokollieren das Abstimmen einer Größe der Warteschlange umfasst, um Abschattung der Schreibvorgänge des ersten Stroms zu regeln.
  6. Verfahren nach Anspruch 5, ferner umfassend das Verwenden von Inhalt, der aus dem Protokollieren stammt, um Teilergebnisse von durch Stromverlust unterbrochenen Datendienstberechnungen zu vervollständigen.
  7. Computersystem (100), Folgendes umfassend:
    einen Anwendungsserver (110), wenigstens einen Prozessor (114) und einen ersten nicht flüchtigen Speicher (124-1) umfassend,
    wobei das System ferner dadurch gekennzeichnet ist, dass der Anwendungsserver (110) angepasst ist zum:
    Erzeugen eines ersten Stroms von Schreibvorgängen in den ersten nicht flüchtigen Speicher (124-1), wobei der erste Strom wenigstens einen Synchronisierungsbefehl zum Erzeugen eines Synchronisierungspunkts für Datenspeicherung umfasst;
    Koordinieren des Replizierens von im ersten nicht flüchtigen Speicher gespeicherten Daten, wobei das Koordinieren als Reaktion auf den ersten Strom von Schreibvorgängen Folgendes enthält: Erzeugen (402) eines zweiten Stroms von Schreibvorgängen für ein zweites nicht flüchtiges Speichersystem (124-2); und
    Einbetten (404) wenigstens eines Synchronisierungsbefehls in den zweiten Strom von Schreibvorgängen zum Erzeugen eines Synchronisierungspunkts für Datenspeicherung im zweiten nicht flüchtigen Speichersystem (124-2) als Reaktion auf den wenigstens einen Synchronisierungsbefehl des ersten Stroms.
  8. Computersystem (100) nach Anspruch 7, ferner Folgendes umfassend:
    einen Speicherserver (146), der einen zweiten nicht flüchtigen Speicher (124-2) zum Speichern von Daten als Reaktion auf den zweiten Strom von Schreibvorgängen und einen dritten nicht flüchtigen Speicher (124-3) umfasst, wobei der Speicherserver (146) angepasst ist, die im zweiten nicht flüchtigen Speicher (124-2) gespeicherten Daten zu duplizieren, um redundante Daten im dritten nicht flüchtigen Speicher (124-3) zu erzeugen.
  9. Computersystem (100) nach Anspruch 8, wobei der Speicherserver (146) angepasst ist, Paritätsinformationen mit den im dritten nicht flüchtigen Speicher (124-3) gespeicherten Daten zu erzeugen und zu speichern.
  10. Computersystem (100) nach Anspruch 8, wobei der Speicherserver (146) angepasst ist, wenigstens einen Wiederherstellungspunkt, der dem wenigstens einen Synchronisierungsbefehl des ersten Stroms zugeordnet ist, zum Anwendungsserver (110) zu bestätigen.
  11. Computersystem (100) nach Anspruch 8, wobei der Speicherserver (146) mehrere physische Maschinen umfasst.
  12. Computersystem (100) nach Anspruch 8, wobei der Speicherserver (146) angepasst ist, einen Hintergrundprozess zu verwenden, um einen Datendienst an im zweiten nicht flüchtigen Speicher (124-2) gespeicherten Daten durchzuführen und eine Priorität des Hintergrundprozesses basierend wenigstens teilweise darauf, ob der Anwendungsserver (110) eine Zuordnung zu entsprechenden im ersten nicht flüchtigen Speicher (124-1) gespeicherten Daten aufweist, zu regeln.
  13. Computersystem (100) nach Anspruch 8, wobei der Speicherserver (146) ferner Folgendes umfasst:
    eine Warteschlange zum Speichern eines auf den zweiten Strom hinweisenden Protokolls.
  14. Artikel, umfassend ein nicht transitorisches computerlesbares Speichermedium zum Speichern von Anweisungen, die, wenn sie durch ein prozessorbasiertes System ausgeführt werden, bewirken, dass das prozessorbasierte System Folgendes durchführt:
    als Reaktion auf einen ersten Strom von Schreibvorgängen in ein erstes nicht flüchtiges Speichersystem (124-1), Erzeugen (402) eines zweiten Stroms von Schreibvorgängen für ein zweites nicht flüchtiges Speichersystem; und
    Koordinieren des Replizierens von im ersten nicht flüchtigen Speichersystem (124-1) gespeicherten Daten, wobei das Koordinieren das Einbetten (404) wenigstens eines Befehls in den zweiten Strom von Schreibvorgängen zum Erzeugen eines Synchronisierungspunkts für Datenspeicherung im zweiten nicht flüchtigen Speichersystem (124-2) als Reaktion auf einen entsprechenden Befehl im ersten Strom zum Erzeugen eines Synchronisierungspunkts für Datenspeicherung im ersten nicht flüchtigen Speichersystem (124-1) umfasst.
  15. Artikel nach Anspruch 14, wobei:
    das erste nicht flüchtige Speichersystem (124-1) nicht flüchtige Speichergeräte umfasst, die aus phasenwechselnden Speichergeräten, Memristor-Speichergeräten und magnetoresistiven Speichergeräten ausgewählt sind; und
    der wenigstens eine Synchronisierungsbefehl des ersten Stroms von Schreibvorgängen verwendet wird, um zu bestätigen, dass Daten in einer Prozessor-Pipeline in dem ersten nicht flüchtigen Speichersystem (124-1) eingespeichert worden sind.
EP13879729.5A 2013-03-28 2013-03-28 Koordination der replikation von in einem nichtflüchtigen speichersystem gespeicherten daten Active EP2979191B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2013/034245 WO2014158155A1 (en) 2013-03-28 2013-03-28 Coordinating replication of data stored in a non-volatile memory-based system

Publications (3)

Publication Number Publication Date
EP2979191A1 EP2979191A1 (de) 2016-02-03
EP2979191A4 EP2979191A4 (de) 2016-10-12
EP2979191B1 true EP2979191B1 (de) 2018-06-27

Family

ID=51624940

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13879729.5A Active EP2979191B1 (de) 2013-03-28 2013-03-28 Koordination der replikation von in einem nichtflüchtigen speichersystem gespeicherten daten

Country Status (4)

Country Link
US (1) US9678673B2 (de)
EP (1) EP2979191B1 (de)
CN (1) CN104937576B (de)
WO (1) WO2014158155A1 (de)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9946489B2 (en) * 2016-07-27 2018-04-17 Western Digital Technologies, Inc. Multi-stream journaled replay
CN106598730B (zh) * 2016-11-24 2020-06-12 上海交通大学 基于非易失性内存的可在线恢复对象分配器设计方法
US10636495B2 (en) 2018-06-12 2020-04-28 Western Digital Technologies, Inc. Adjustable read retry order based on decoding success trend
US11853758B2 (en) * 2019-09-27 2023-12-26 Intel Corporation Techniques for decoupled access-execute near-memory processing
US11561717B2 (en) * 2020-05-26 2023-01-24 Western Digital Technologies, Inc. Data integrity protection of SSDs utilizing streams

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7039663B1 (en) * 2002-04-19 2006-05-02 Network Appliance, Inc. System and method for checkpointing and restarting an asynchronous transfer of data between a source and destination snapshot
US7437521B1 (en) * 2003-08-18 2008-10-14 Cray Inc. Multistream processing memory-and barrier-synchronization method and apparatus
US7143122B2 (en) * 2003-10-28 2006-11-28 Pillar Data Systems, Inc. Data replication in data storage systems
US7188273B2 (en) 2003-11-24 2007-03-06 Tsx Inc. System and method for failover
US8078813B2 (en) * 2004-09-30 2011-12-13 Emc Corporation Triangular asynchronous replication
JP2007086972A (ja) * 2005-09-21 2007-04-05 Hitachi Ltd ストレージシステム、二重化制御方法、及びプログラム
JP4813924B2 (ja) * 2006-02-28 2011-11-09 株式会社日立製作所 データベース管理システム、ストレージ装置、ディザスタリカバリシステム及びデータベースのバックアップ方法
US7809892B1 (en) 2006-04-03 2010-10-05 American Megatrends Inc. Asynchronous data replication
US7627729B2 (en) 2006-09-07 2009-12-01 International Business Machines Corporation Apparatus, system, and method for an improved synchronous mirror swap
JP5057366B2 (ja) * 2006-10-30 2012-10-24 株式会社日立製作所 情報システム及び情報システムのデータ転送方法
US20080162728A1 (en) 2007-01-03 2008-07-03 Microsoft Corporation Synchronization protocol for loosely coupled devices
US7822892B2 (en) 2007-07-19 2010-10-26 International Business Machines Corporation Managing the copying of writes from primary storages to secondary storages across different networks
US7882388B2 (en) * 2008-08-21 2011-02-01 Sierra Wireless America, Inc. Dual independent non volatile memory systems
US8438332B2 (en) 2009-06-22 2013-05-07 International Business Machines Corporation Apparatus and method to maintain write operation atomicity where a data transfer operation crosses a data storage medium track boundary
US9235595B2 (en) 2009-10-02 2016-01-12 Symantec Corporation Storage replication systems and methods
US8145945B2 (en) 2010-01-04 2012-03-27 Avaya Inc. Packet mirroring between primary and secondary virtualized software images for improved system failover performance
CN101808137B (zh) 2010-03-29 2014-09-03 成都市华为赛门铁克科技有限公司 数据传输方法、装置和系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
US20150317079A1 (en) 2015-11-05
EP2979191A1 (de) 2016-02-03
CN104937576B (zh) 2018-08-31
CN104937576A (zh) 2015-09-23
EP2979191A4 (de) 2016-10-12
US9678673B2 (en) 2017-06-13
WO2014158155A1 (en) 2014-10-02

Similar Documents

Publication Publication Date Title
US10152381B1 (en) Using storage defragmentation function to facilitate system checkpoint
US9781227B2 (en) Lockless distributed redundant storage and NVRAM caching of compressed data in a highly-distributed shared topology with direct memory access capable interconnect
TWI709073B (zh) 分佈式儲存系統、分佈式儲存方法及分佈式設施
US10133662B2 (en) Systems, methods, and interfaces for managing persistent data of atomic storage operations
US9218278B2 (en) Auto-commit memory
US10817421B2 (en) Persistent data structures
US10817502B2 (en) Persistent memory management
US9047178B2 (en) Auto-commit memory synchronization
US10152267B1 (en) Replication data pull
US8429363B2 (en) Multiple incremental virtual copies
US20140208012A1 (en) Virtual disk replication using log files
US20180095684A1 (en) Replicating tracks from a first storage site to a second and third storage sites
US8825973B2 (en) Caching source blocks of data for target blocks of data
US9262344B2 (en) Local locking in a bi-directional synchronous mirroring environment
WO2015020811A1 (en) Persistent data structures
US8862819B2 (en) Log structure array
EP2979191B1 (de) Koordination der replikation von in einem nichtflüchtigen speichersystem gespeicherten daten
US20140344503A1 (en) Methods and apparatus for atomic write processing
US10067882B2 (en) Storage system and storage control method
US11650920B1 (en) Write cache management
US11315028B2 (en) Method and apparatus for increasing the accuracy of predicting future IO operations on a storage system
US10740189B2 (en) Distributed storage system
WO2018055686A1 (ja) 情報処理システム
US11966590B2 (en) Persistent memory with cache coherent interconnect interface
US20190205044A1 (en) Device for restoring lost data due to failure of storage drive

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20150728

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: HEWLETT PACKARD ENTERPRISE DEVELOPMENT L.P.

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20160914

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 3/06 20060101ALI20160908BHEP

Ipc: G06F 11/20 20060101ALI20160908BHEP

Ipc: G06F 13/14 20060101AFI20160908BHEP

Ipc: G06F 9/06 20060101ALI20160908BHEP

Ipc: G06F 9/30 20060101ALI20160908BHEP

Ipc: G06F 12/02 20060101ALI20160908BHEP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602013039566

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06F0013140000

Ipc: G06F0011140000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 9/06 20060101ALI20180118BHEP

Ipc: G06F 12/02 20060101ALI20180118BHEP

Ipc: G06F 11/20 20060101ALI20180118BHEP

Ipc: G06F 9/30 20180101ALI20180118BHEP

Ipc: G06F 3/06 20060101ALI20180118BHEP

Ipc: G06F 13/14 20060101ALI20180118BHEP

Ipc: G06F 11/14 20060101AFI20180118BHEP

INTG Intention to grant announced

Effective date: 20180212

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1012912

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602013039566

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180927

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180927

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20180627

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180928

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1012912

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180627

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181027

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602013039566

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20190220

Year of fee payment: 7

26N No opposition filed

Effective date: 20190328

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190328

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190331

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190328

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181029

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190328

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20130328

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180627

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602013039566

Country of ref document: DE

Owner name: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP, SPR, US

Free format text: FORMER OWNER: HEWLETT PACKARD ENTERPRISE DEVELOPMENT LP, HOUSTON, TEX., US

Ref country code: DE

Ref legal event code: R082

Ref document number: 602013039566

Country of ref document: DE

Representative=s name: MARKS & CLERK (LUXEMBOURG) LLP, LU

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230321

Year of fee payment: 11

Ref country code: DE

Payment date: 20230328

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240328

Year of fee payment: 12

Ref country code: GB

Payment date: 20240319

Year of fee payment: 12