JP2019128959A - Key-value data reliability system, data storage method therefor, and non-transitory computer-readable medium including computer code implementing that method - Google Patents

Abstract

To provide a method of storing data in a key-value reliability system including N storage devices, where N is an integer, grouped into a reliability group as a single logical unit and managed by a virtual device management layer.SOLUTION: A method of storing data in a key-value reliability system including N storage devices, where N is an integer, grouped into a reliability group as a single logical unit and managed by a virtual device management layer includes determining whether the data meets a threshold corresponding to a reliability mechanism for storing the data, selecting the reliability mechanism when the data satisfies the threshold, and storing the data according to the selected reliability mechanism.SELECTED DRAWING: Figure 1

Description

  The disclosure according to the present invention (hereinafter also simply referred to as “the present invention”) generally relates to a data storage system, and more particularly, to a key value reliability system including a plurality of key value storage devices. It is related with the method of selecting the reliability mechanism which can store.

  Data reliability mechanisms such as erasure coding solve data loss due to storage device failure and 'data corruptions' in various facilities including multiple storage devices. Can be used for

Conventional solid state drives (SSDs) generally use only a block interface, and through erasure array of independent disks (RAID), erase coding or replication Can provide data reliability.
Since the object formats have various sizes and are not structured, efficient data conversion is required at the object-block level interface. Furthermore, there is a need to ensure data reliability while maintaining space efficiency and fast access time characteristics.

  Techniques such as RAID are widely studied for conventional block storage devices. However, relatively new key-value storage devices have different interfaces and different storage semantics than conventional block devices. Therefore, for a variety of new key-value storage devices, potential new data reliability techniques that are custom-made or specifically adopted for key-value data and key-value storage devices may be useful There is sex.

U.S. Patent No. 9,594,633 U.S. Patent No. 9,639,268

  An object of the present invention is data of a key-value reliability system including N (where N is an integer) storage devices grouped into one reliability group as a single logical unit and managed by a virtual device management layer It is to provide a storage method.

  Embodiments in accordance with the present invention each perform a single key recovery procedure that allows the reliability mechanism to recover all keys present in the failed memory device and allow the virtual device management layer to copy to the new memory device. As it can, the embodiments described herein can contribute to improvements in the memory storage field.

(1) According to an embodiment of the present invention, it includes N (where N is an integer) storage devices grouped into one reliability group as a single logical unit and managed by the virtual device management layer A data storage method of a key-value reliability system is provided.
The method determining whether the data satisfies a threshold corresponding to a reliability mechanism for storing the data, selecting the reliability mechanism if the data is satisfied the threshold; And storing the data in response to the selected reliability mechanism.

(2) The threshold may be an object size of the data, a throughput consideration of the data, a read / write temperature of the data (i.e., a read / write frequency), and the N Based on one or more of the storage device's underlying erasure coding capabilities.
(3) The method further includes testing the data against the reliability mechanism using one or more Bloom filters or caches.
(4) The selected reliability mechanism, one or more checksums for each of the N storage devices storing the data, stored in each of the N storage devices storing the data Metadata for recording the size of the value object of the data and the position of the parity group member of the N storage devices indicating where the data is stored in the N storage devices. The method may further include inserting with the key corresponding to the data.
(5) The selected reliability mechanism includes object replication, and storing the data includes selecting a KV value, and calculating a hash for hashing a key corresponding to the selected KV value. Determining a subset of storage devices among the N storage devices to store a replica of the key object corresponding to the KV value, and under the same user key name Writing the updated value corresponding to the KV value into each of the storage device subsets.

(6) The selected reliability mechanism includes packing, and the step of storing the data is performed by selecting k among the N storage devices of the grouped reliability group, where k is Selecting k key objects stored in an integer number of storage devices, collecting k value objects corresponding to the k key objects, a virtual size of all the k value objects being Padding virtual zero to the end of the value object that does not have the maximum value size among the k value objects, so that r (where r is an integer) from the k key objects Generating parity objects of the plurality of storage devices, writing the k key objects to the k storage devices, and generating the r parity objects of the N storage devices. In comprising the step of writing to the r storage device, wherein each of r pieces of storage device is distinguished as the k-number of the storage device (distinct), provided that k + r = N.

(7) The selected reliability mechanism includes packing using a traditional erasure coding, and the N storage devices have a traditional (k, r) maximum distance separate (MDS) erasure coding. Configured
(8) The selected reliability mechanism includes packing using regeneration erasure coding, and the N storage devices are configured with (k, r, d) regeneration erasure coding.

(9) The selected reliability mechanism includes splitting, and storing the data comprises selecting a KV value, and dividing the KV value into k (where k is an integer) equal sized objects Generating r (where r is an integer) parity objects from the k equal-sized objects, calculating a hash for hashing a key corresponding to the selected KV value, Determining a main storage device where the KV value is located among the N storage devices based on a hash, and the k equal-sized objects and the r pieces in a sequential order starting from the main storage device Writing each of the N parity objects to the N storage devices, where k + r = N.

(10) The selected reliability mechanism includes splitting using traditional erasure coding, and the N storage devices are configured with traditional (k, r) maximum distance separate (MDS) erasure coding. .
(11) The selected reliability mechanism includes packing using regenerative erasure coding, and the N storage devices are configured with (k, r, d) regenerative erasure coding and storing the data. Divides the k identically sized value objects into m (where m is an integer) subpackets using the regenerative erasure coding, and each of the r parity objects is m The method further includes dividing into parity subpackets.

(12) According to another embodiment of the present invention, there is provided a data reliability system for storing data based on a selected reliability mechanism. The data reliability system is N (where N is an integer) storage configured as a virtual device using dataless data protection in a data reliability system storing data based on a selected reliability mechanism According to the device and the selected reliability mechanism, the N storage devices are managed as the virtual device and configured to store data in a storage device selected among the N storage devices. The virtual device management layer, wherein the virtual device management layer determines whether the data meets a threshold corresponding to a reliability mechanism for storing the data, the threshold being satisfied by the data. If configured, the reliability mechanism is selected to store the data according to the selected reliability mechanism. That.

(13) The selected reliability mechanism includes object replication, and the virtual device management layer selects a KV value, calculates a hash for hashing a key corresponding to the selected KV value, and the KV A subset of storage devices is determined among the N storage devices to store replicas of key objects corresponding to values, and a subset of the determined storage devices is stored under the same user key name. Are each configured to store the data by writing updated values corresponding to the KV values.
(14) The selected reliability mechanism includes packing, and the virtual device management layer includes k keys stored in k (where k is an integer) storage devices among the N storage devices. An object is selected, k value objects corresponding to the k key objects are collected, and the virtual value size of all the k value objects is the same among the k value objects. A virtual zero is padded to the end of the value object that does not have the maximum value size, and r (where r is an integer) parity objects are generated from the k key objects, and the k key objects are defined as the k objects. The storage device is configured to store the data by writing to the storage devices and writing the r parity objects to the r storage devices among the N storage devices. Each of the r storage devices is distinguished from each of the k storage devices, where k + r = N.

(15) The selected reliability mechanism includes splitting, the virtual device management layer selects a KV value, and divides the KV value into k (where k is an integer) equal sized value objects, R parity objects are generated from the k same size value objects, where r is an integer, calculates a hash for hashing a key corresponding to the selected KV value, based on the hash A primary storage device in which the KV value is located is determined from among the N storage devices, and each of the k value objects of the same size and the r parity objects is sequentially arranged starting from the primary storage device. Are configured to store the data by writing to the N storage devices, where k + r = N It is.
(16) The selected reliability mechanism includes splitting using regeneration and erasure coding, the N storage devices are configured with (k, r, d) regeneration and erasure coding, and the virtual device management layer is Using the regenerative erasure coding, the k objects of the same size are divided into m (where m is an integer) subpackets, and each of the r parity objects is divided into m parity subpackets. The division is further configured to store the data.

(17) According to another embodiment of the present invention, N storages grouped into one reliability group as a single logical unit and managed by the virtual device management layer when executed on a processor A non-transitory computer readable medium comprising computer code embodying a method of storing data in a key-value reliability system comprising the device is provided. The method determining whether the data satisfies a threshold corresponding to a reliability mechanism for storing the data, selecting the reliability mechanism if the data is satisfied the threshold; And storing the data according to the selected reliability mechanism.

(18) The selected reliability mechanism includes object replication, and storing the data includes selecting a KV value, and calculating a hash for hashing a key corresponding to the selected KV value. Selecting a subset of storage devices among the N storage devices to store replicas of key objects corresponding to the KV values, and under the same user key name, to the KV values. Writing the corresponding updated value into each of the determined storage device subsets.
(19) The selected reliability mechanism includes packing, and the step of storing the data is stored in k (where k is an integer) storage devices among the N storage devices of the reliability group. Selecting k key objects, collecting k value objects corresponding to the k key objects, and setting the k value objects such that the virtual value sizes of all the k value objects are the same. Padding virtual zeros at the end of the value objects not having the largest value size among the value objects, generating r (where r is an integer) parity objects from the k key objects, Writing the k key objects to the k storage devices, and writing the r parity objects to r storage devices among the N storage devices. Each of the r storage devices is divided into the k storage devices, and k + r = N.
(20) The selected reliability mechanism includes splitting, and storing the data comprises selecting a KV value, and dividing the KV value into k (where k is an integer) equal sized value objects Generating r (where r is an integer) parity objects from the k equal-sized value objects, and calculating a hash for hashing a key corresponding to the selected KV value. Selecting the primary storage device in which the KV value is located among the N storage devices based on the hash, and the k identically sized values in sequential order starting from the primary storage device. Writing each of the object and the r parity objects into the N storage devices, where k + r = N is there.

  According to the reliability mechanism according to the embodiment of the present invention, the virtual device management layer performs each single key recovery procedure, which recovers all keys present in the failed memory device and enables copying to the new memory device. Therefore, the reliability of the memory storage can be improved.

The above-described idea and / or other ideas will become clearer and more fully appreciated from the detailed description of the following embodiments in combination with the accompanying drawings.
1 is a block diagram illustrating a key-value reliability system that stores key-value data based on selected reliability mechanisms according to an embodiment of the present invention. FIG. FIG. 6 is a flow chart illustrating selection of a reliability mechanism used by a key-value reliability system based on a size threshold corresponding to the size of data of a key-value pair according to an embodiment of the present invention. According to an embodiment of the present invention, configured to store key-value data according to a reliability mechanism of K-object (k, r) erasure coding using multi-object "packing" using traditional erasure coding. FIG. 7 is a block diagram showing a group of KV storage devices. FIG. 4 is a block diagram illustrating storage of value objects and parity objects according to a reliability mechanism of K-object (k, r) erasure coding using multiple erasure coding, that is, multi-object “packing”, according to an embodiment of the present invention. It is. Key-value data according to the reliability mechanism of “Single Object (k, r) erasure coding”, or “Splitting”, using traditional erasure coding according to an embodiment of the present invention. 2 is a block diagram showing a group of KV storage devices configured to store the. Key value depending on the reliability mechanism of “Single Object (k, r, d) erasure coding”, ie, “Splitting”, using regenerative erasure coding according to an embodiment of the present invention. FIG. 5 is a block diagram illustrating a group of KV storage devices configured to store data.

  Various features of the present invention and methods of achieving the same may be understood in more detail with reference to the accompanying drawings and the following detailed description of the embodiments. Embodiments are described in more detail below with reference to the attached drawings. Similar reference numbers generally refer to similar components. However, it should not be understood that the present invention can be embodied in various other forms and is not limited to the embodiments illustrated herein. Rather, such embodiments are provided as an illustration to assist in a general understanding of the invention, and may sufficiently convey the technical features and aspects of the invention to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary for those skilled in the art to fully understand the features and aspects of the present invention may not be described. Unless otherwise stated, like reference numerals refer to similar elements throughout the attached drawings and the detailed description set forth, and the description thereof will not be repeated. In the drawings, the relative sizes of elements, hierarchies, and regions can be exaggerated for clarity.

  Various embodiments are described herein with reference to cross-section illustrations that are schematic illustrations of embodiments and / or intermediate structures. Thus, for example, variations from the illustrated form as a result of manufacturing techniques and / or tolerances should be expected. Furthermore, the specific structural or functional description given in the text is a simple illustration to describe an embodiment according to the inventive idea. That is, the embodiments described herein should not be construed as being limited to the illustrated form of identifying a region, but should include, for example, shape deviations that are the result of manufacturing. For example, rectangularly illustrated implant regions are traditionally rounded at their edges, rather than binary changes from implants to non-implant regions, or have curved features and / or gradients of implant concentration. Can do. Similarly, the implant area formed by the implant can trigger some implants in the area between the implant area and the surface on which the implant occurs. That is, the areas illustrated in the drawings are essentially schematic and their shape is not intended to illustrate the actual shape of the area of the device, and is not limiting. Additionally, as one of ordinary skill in the art can appreciate, the described embodiments can be modified in various other ways without departing from the spirit or scope of the present invention.

  In the following detailed description, numerous specific details are provided to aid in the convenience of the description and the understanding of the various embodiments. However, various embodiments can be embodied without a detailed description or with one or more equivalent alternatives. In other instances, widely known structures and devices are shown in block diagram form in order not to obscure the various embodiments unnecessarily.

  Terms such as “first,” “second,” “third,” etc. may be used in the text to describe various elements, configurations, areas, hierarchies, and / or areas. It should be understood that although used, such elements, configurations, areas, hierarchies, and / or areas are not limited to such terms. Such terms are only used to separate one element, configuration, area, hierarchy or area from another element, configuration, area, hierarchy or area. That is, a first element, configuration, region, hierarchy, or area described below may be referred to as a second element, configuration, region, hierarchy, or area without departing from the spirit and scope of the present invention. .

  Spatial relative terms such as “beath, below, lower, under”, “above, upper”, etc. are other ones illustrated in the drawings. It can be used herein to easily describe the association of an element or feature with an element or feature. It is well understood that spatially relative terms are intended to include other orientations of the device in operation or use in addition to the directivity illustrated in the drawings. For example, if a device is flipped in the drawing, an element described as “below or beneath or under” other elements or features will go “above” other elements or features. . That is, the exemplary term “below, under” can include all up and down directions. The device can be oriented in other directions (eg, rotated 90 ° or other directions), and the spatially relative description used in the text should be interpreted accordingly. Similarly, if it is described that the first part is aligned with the second part “on”, this is based on the direction of gravity, with no limitation on its upper surface, and the first part is the upper part of the second part. Indicates that it is aligned to the surface or lower surface.

  When an element, hierarchy, region, or configuration is referred to as "on, connected to, or coupled to" with another element, hierarchy, region, or configuration, the other component, hierarchy, region, or Or, there may be one or more intermediate elements, hierarchies, regions, or configurations that are directly connected to the configuration. However, the term "directly linked" refers to one component being directly linked to another without intermediate construction. On the other hand, other expressions that describe the relationship between constructs, such as "between, immediately between" or "adjacent to or directly adjacent to" can be interpreted similarly. , Where an element or hierarchy is referred to as being between two elements or hierarchies, only an element or hierarchy exists between the elements or configurations, or one or more intermediate elements or It can be understood that more hierarchies can exist.

  The terminology used herein is merely illustrative for describing a particular embodiment and is not intended to limit the invention. As used herein, the singular term is intended to include the plural forms unless the context clearly indicates otherwise. When the term "including" is used in the detailed description, it defines the presence of the recited feature, an integer, a step, an action, an element, and / or a configuration, but one or more other features, an integer Does not exclude the presence or addition of steps, acts, elements, configurations, and / or groups thereof. As used herein, the term “and / or” includes any union or part of one or more of the listed listed related.

  As used herein, the terms "substantial", "about, approximately" and similar terms are used as approximate terms and as terms of degree Rather, it is intended to describe the inherent deviation with measured or calculated values that can be recognized by those skilled in the art. As used herein, the term “about” includes the stated value, taking into account the error associated with the particular amount of measurement and the measurement in question (eg, measurement system limitations), It is meant to be within an acceptable deviation for a particular value determined by one skilled in the art. For example, "about" means within one or more standard deviations, or within ± 30%, 20%, 10%, 5% of the stated value it can. As used herein, the term "use, using, and used" may be considered synonymous with "utilize, utilizing, and utilized". Also, the term “exemplary” is intended to refer to “example or illustration”.

  If a particular embodiment is implemented differently, a particular process order may be performed differently than the order described. For example, two consecutively described processes can be performed substantially simultaneously or in an order opposite to the order described.

  The electrical or electronic devices and / or other associated devices or configurations according to embodiments of the invention described herein are suitable hardware, firmware (e.g., application-specific integrated circuit (ASIC), application-specific integrated circuit). , Software, or a combination of software, firmware, and hardware, for example, the various configurations of such devices may be integrated circuits (ICs) chips or other In addition, various configurations of such devices include flexible printed circuit films, tape carrier packages (TCP; ta e carrier package), printed circuit board (PCB) may be embodied, or may be formed on a single substrate In addition, various configurations of such devices are described herein Processes or threads driven by one or more processors in one or more computing devices that communicate with other system configurations and execute computer program instructions to perform the various functions described above The computer program instructions are stored in a memory that can be embodied in a computing device using a standard memory device such as a RAM (Random Access Memory), such as a CD-ROM, It can be stored on other non-transitory computer readable media, such as a rush drive, etc. Also, those skilled in the art will recognize a variety of without departing from the spirit and aspect of the exemplary embodiments of the invention. Understand that the functions of a computing device can be combined or integrated into a single computing device, or that the functions of a particular computing device are distributed to one or more other computing devices Can do.

  Unless otherwise defined, all terms including technical / scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in the shared dictionary should be construed as having a meaning consistent with their meaning in the context of the relevant art and / or in the detailed description of the invention, and are explicitly stated in the text Unless otherwise defined, it should not be interpreted as ideal or too formal.

  As described below, embodiments of the disclosed invention are a key-value reliability system (key-value) comprised of multiple key-value (KV) storage devices grouped into a single logical unit. provides a method for storing key-value data in a reliable manner. Furthermore, embodiments of the present invention provide a stateless hybrid reliability manager that manages drives and controls storage of key-value (KV) pairs. The stateless hybrid trust manager is responsible for object replication; K-object (k, r) erasure coding-packing (K-Object (k, r) erasure coding-packing); single object (k, r). Erase coding-Splitting (Single Object (k, r) erasure coding-Splitting); K-Object (k, r, d) Regenerating coding-Packing (K-Object (k, r, d) regeneration coding-Packing); Multiple objects including single object (k, r, d) regenerating coding-splitting (Single Object (k, r, d) regeneration coding-splitting) Depend on the pluggable reliability mechanism / technique / implementation.

  A stateless hybrid reliability manager dependent on multiple pluggable reliability mechanisms can manage the device, control the storage of KV pairs, and the disclosed method for selection of reliability mechanisms has different sized KV pairs As efficient storage, retrieval, and recovery can be ensured, the described embodiments can improve the performance of memory storage (eg, storage of key-value data in a key-value storage device).

  FIG. 1 is a block diagram illustrating a key-value reliability system that stores key-value data based on a selected reliability mechanism according to an embodiment of the present invention.

Referring to FIG. 1, as described above, key-value data and various other key-value (KV) “storage devices” {alias “memory devices” “drives” “KV-SSD”} (130) New data reliability mechanisms that are tailored or specifically adopted for the KV storage device 130 may be ubiquitous. Thus, a hybrid key-value reliability system for such a KV storage device 130 uses the KV storage device 130 according to one or more pluggable reliability mechanisms. And a stateless hybrid reliability manager / virtual device manager layer / virtual device management layer 120 that controls storage of KV pairs to them.
Even though solid-state drives (SSDs) are commonly used to refer to the KV storage devices described herein, other storage devices may be used in accordance with embodiments of the present invention. The design and operation of virtual device management layer 120 according to an embodiment of the present invention is described below.

  In the current embodiment, the virtual device management layer 120 may enable a method for storing the key value data / KV pair 170 reliably in the key value reliability system. The key-value reliability system includes multiple KV storage devices 130 grouped into a single logical unit. A single logical unit is referred to as a reliability group 140.

  The KV storage device 130 of the reliability group 140 stores each chunk of erase-coded data and / or replicated data corresponding to the key-value data 170. The KV storage device 130 of the reliability group 140 is apparently exposed as a single virtual device 110 with key-value operations being directed to the virtual device management layer 120.

  The virtual device 110 includes a stateless hybrid reliability manager as a virtual device management layer 120. That is, the virtual device management layer 120 operates in a stateless manner (i.e., it is not necessary to maintain the mapping between key values and devices).

  Accordingly, the virtual device 110 has a key value through N KV storage devices 130 (where N is an integer) (for example, KV-SSD 130-1, 130-2, 130-3, 130-4,..., 130-N). Data 170 is stored, and key value data 170 is stored in the KV storage device 130 through the virtual device management layer 120. In other words, the virtual device management layer 120 manages the KV storage device 130 and controls storing KV pairs in it.

  In another embodiment, the key-value reliability system may further include a cache that selectively stores metadata and / or data associated with keys of the key-value data 170 to improve operation speed. The reliability mechanism can attach the value of the KV pair to include metadata corresponding to the KV pair. That is, both the key and the value can be attached with information corresponding to the metadata identifier “MetaID” to store additional metadata specific to the KV pair. The metadata includes a checksum, a reliability mechanism identifier for identifying a reliability mechanism used to store data, an erasure code identifier, an object size, a position of a parity group number, and the like.

  In other embodiments, the key-value trust system may further include bloom filters that correspond to the trust mechanisms described below. The Bloom filter can store the stored key using a corresponding reliability mechanism, which can support the key-value reliability system in the read operation. Thus, one or more bloom filters or caches of a key-value trust system can allow rapid testing of keys against existing trust mechanisms.

  Each of the reliability mechanisms described herein uses the same hash function for a single numeric key modulo in the KV storage device 130 using the first hash function corresponding to the key-value data 170. A copy or chunk of can be stored first. That is, for each pluggable trust mechanism, the trust mechanism can store at least the first copy / chunk using the same key as the user key.

  As mentioned above, embodiments of the present invention provide multiple pluggable reliability mechanisms that ensure reliable storage of key-value data 170 in multiple KV storage devices 130. Accordingly, the virtual device management layer 120 requires a reliability mechanism and can determine which one is used in the reliability mechanism.

  The reliability mechanism may be based on policies such as value-size thresholds and / or object read / write frequency set during setup of the virtual device 110. Thus, the virtual device management layer 120 can select an appropriate reliability mechanism based on the specified policies of the system.

  For the five reliability mechanisms of the embodiment of the present invention, how the reliability mechanism operates by the virtual device management layer 120 and when the reliability mechanism is properly used and selected are described below. Describe. These five reliability mechanisms are object replication, K object (k, r) erasure coding-packing (K-Object (k, r) erasure coding-packing), single object (k, r) erasure. Coding-Splitting (Single Object (k, r) erasure coding-Splitting), K Object (k, r, d) Regeneration Coding-Packing (K-Object (k, r, d) regeneration coding-Packing), and Single One object (k, r, d) may be referred to as regenerating coding (single object (k, r, d) regeneration coding).

  FIG. 2 is a flow diagram (200) illustrating the selection of a reliability mechanism used by a key-value reliability system based on a size threshold corresponding to the size of data of a KV pair according to an embodiment of the present invention.

Referring to FIG. 2, for an entire supported reliability mechanism based on a size threshold (eg, the five above-described reliability mechanisms), the virtual device management layer 120 has data (eg, key-value data 170). A first reliability mechanism that satisfies the requirement of the value size threshold, and can determine whether the value size of the value size is smaller than a given threshold (t _i ) corresponding to each reliability mechanism. It can be selected.

For example, in S210, the virtual device management layer 120 receives “n” (n is an integer) reliability mechanisms supported on a size threshold basis. At S220, the virtual device management layer 120 simply considers each of the reliability mechanisms, one at a time in the order 1 to n. At S230, at each discussion of the reliability mechanisms to be supported, the virtual device management layer 120 determines whether the value size of the data is smaller than the threshold t _i corresponding to each reliability mechanism.

If at S240 a reliability mechanism having a threshold t _i greater than or equal to the value size of the data is found, the virtual device management layer 120 chooses to use that reliability mechanism. In S250, determination and reliability mechanisms with appropriate thresholds t _i satisfying Reliability mechanisms is determined, or S220 of the value size in the final reaction birefringence used in S240 is not within the N Reliability Mechanism If so, the virtual device management layer 120 ends the determination of the reliability mechanism to be used.

  In the current embodiment, “n” is equal to 5 depending on the five different reliability mechanisms of the embodiments described herein. For relatively small key values (i.e., relatively small value sizes), the virtual device management layer 120 chooses to use the Object Replication reliability mechanism. For slightly larger key values, the virtual device management layer 120 chooses the reliability mechanism for packing and then splitting (eg, in order), while using traditional erasure coding for each To do. However, for larger key values, the virtual device management layer 120 selects packing and then splitting, while using regeneration erasure coding instead of traditional erasure coding. .

  In an embodiment of the present invention, the selection of a reliability mechanism for use includes object size, processing capacity requirement for the object, read / write temperature of the corresponding key-value pair, ie read / write temperature. It may be based on one or more of write frequency, basic 8 underlying of multiple KV storage devices, and / or detection of whether the key is hot or cold. For example, while "hot" keys use the object replication reliability mechanism regardless of their value size, "cold" keys depend on their value size. One of the reliability mechanisms of the erasure coding scheme is applied. In another embodiment, the decision on whether or not to use an object replication reliability mechanism may be based on both size and read / write temperature. Thus, the object read / write temperature, ie, the threshold corresponding to the object read / write frequency, may be used to determine the reliability mechanism instead of the threshold corresponding to the size in the sequence diagram 200 of FIG.

  The operation of each of the five reliability mechanisms is described below.

  Referring back to FIG. 1, as previously mentioned, when the value size of the KV pair is relatively small, the reliability mechanism of “Object Replication” is appropriate as a choice by the virtual device management layer 120 is there. Object replication is applied to each object (eg, each key value data / KV pair 170). The reliability mechanism of Object Replication has high storage overhead but is suitable for very small value sizes because of the low read and recovery costs.

  The object replication reliability mechanism is also appropriate for key-values with frequent updates (eg, key-value data 170) and is therefore selected based on read and write frequencies.

  Each time writing occurs during object replication, the key value data 170 is replicated to one or more additional KV storage devices 130. A primary copy of the key-value data 170 is placed in one of the KV storage devices 130 designated by a key modulo (N) hash. Replicas (replicas) of the main copy of the key-value data 170 are arranged in a sequential manner on the KV storage apparatus 130 or immediately adjacent KV storage apparatus 130 in a cyclic manner.

  The virtual device management layer 120 or the user determines how many copies of the key value data 170 are to be generated. For example, a distributed system using the virtual device management layer 120 can select 3-way replication and set 3-way replication as a default. However, the user of the system can configure the number of object replicas to be greater or less than a selected default.

  Thus, for example, if 3-way replication is used and the primary KV storage device 130-2 contains a primary copy of data (eg, key-value data 170), then the virtual device management layer 120 will be a replica of the primary copy of the data Are stored in the subsequent replica KV storage devices 130-3 and 130-4, and all copies of the data are the same. That is, two (or more) direct following KV storage devices 130-3, 130-4 (eg, in a cyclic fashion) that follow the KV storage device 130-2 that contains the primary copy of the data. Stored in

  A copy of the data is stored in the duplicate KV storage devices 130-3 and 130-4 under the same keyname / same user key as the main KV storage device 130-2. Every copy of the data includes an identifier and a checksum to identify the replicated key value data 170.

  Therefore, when a specific KV storage device 130 has failed (for example, when the KV storage device 130-3 has failed), the KV storage devices 130 immediately before and after the failed KV storage device 130-3 ( For example, the KV storage device 130-3, 130-4, 130-4) directly before and after the KV storage device 130-3) recovers the value by using a recovery mechanism for the key name, thereby reproducing the duplicated key. Ensure that they are restored.

  In summary of the object replication reliability mechanism, the virtual device management layer 120 receives key-value data 170, and KV storage device used to hash key objects and store key object replicas. 130 is determined. The virtual device management layer 120 subsequently selects the updated KV storage device 130 (eg, selected KV storage device 130) under the same user key name (eg, appropriate MetaID field). -2, 130-3, 130-4).

  FIG. 3 shows K-object (k, r) erasure coding (K-Object (k, r) erasure coding) or multi-object “, using traditional erasure coding according to an embodiment of the present invention FIG. 6 is a block diagram illustrating a group of KV storage devices configured to store key-value data according to a reliability mechanism of “multiple object“ Packing ”.

  Referring to FIG. 3, the packing reliability mechanism using traditional erasure coding is a small value that is not suitable to be divided into chunks (e.g. to get better data throughput) Selected for data having a size. For example, a packing reliability mechanism that uses traditional erasure coding is a larger value size (but still relatively smaller than the value size that results in the choice of the Object Replication reliability mechanism described previously. Selected by the virtual device management layer 120.

  Packing using traditional erasure coding is configured with traditional (k, r) MDS (maximum distance separable) erasure coding and used with systematic MDS code. For example, the erasure code is basically (by_default), (4, 2) Reed-Solomon code (Reed-Solomon Code) because (4, 2) Reed-Solomon code has been studied relatively widely. The reason is that the corresponding high speed implementation library is easily available.

  By using a packing reliability mechanism using traditional erasure coding, N different KV storage devices 330-1,..., 330-N that form the same parity group / erasure code group 340. The k key / key objects 350 from the queue of k different KV storage devices 330 that are a part of are selected, erasure coded and packed (k is an integer).

  For example, the virtual device management layer 120 may maintain a buffer of key objects 350 recently written to each KV storage device 330 (eg, each KV storage device 130 of the reliability group 140 of FIG. 1) The device management layer 120 selects k key objects 350 from k different KV storage devices to enable execution of erasure coding, thereby packing k key objects 350 corresponding to KV pairs.

  In the current embodiment, the virtual device management layer 120 selects four key objects 350x, 350y, 350b, 350c from four different KV storage devices 330-1, 330-3, 330-4, 330-N, respectively ( In the current embodiment, k = 4).

  FIG. 4 illustrates the storage of value objects and parity objects according to the reliability mechanism of K-object (k, r) erasure coding, or multi-object “packing” using traditional erasure coding according to an embodiment of the present invention. It is a block diagram shown.

Referring to FIGS. 3 and 4, the key object 350 is positioned in the “key modulo n hash” th KV storage device 330. That is, the hash of each key modulo n is performed for each key object 350 and transmitted to the queue of that particular KV storage device 330. In the current embodiment, the i-th key (Key _i ) 350-i is hashed and located in KV-SSD1 (330-1), the j-th key (Key _j ) 350-j is hashed, and KV-SSD2 ( 330-2), the k th key (Key _k ) 350-k is hashed and located in KV-SSD4 (330-4).

User value length / value size 462 of each stored value object 450 is the same as the written user value length / value size. However, for consistency to allow for erasure coding, user value / value objects 450 are all the same size by having “0” filling / virtual zero / virtual zero padding 464 attached to them. It is considered to have
That is, since the user value size 462 of each of the different value objects 450 may change (that is, the value object 450 has a variable length or is a variable-length key value), virtual zero By embodying the padding 464 method (ie, padding zeros of the virtual zero padding 464 into the value object 450 for coding, while actually rewriting the data representing the value object 450 including the padded zeros. Parity object 460 has the same size as the largest sized value object 470 in parity group 340).
Thus, in the present embodiment, the value objects 450 "Val x", "Val y" and "Val b" are padded with virtual zeros that are not actually stored in the KV storage device, whereby the maximum sized value object It is considered to have the same size as “Val c”. Thus, a parity object 460 is calculated.

  After coding the k key objects 350, the virtual device management layer 120 calculates r parity objects 460 from k values / k value objects 450 corresponding to the k key objects 350. At this time, r is an integer, k + r = N, and N is the number of KV storage devices 330 in the parity group 340 (for example, the N KV storage devices 130 in the reliability group 140 in FIG. 1).

  The virtual device management layer 120 processes the r parity objects 460 to the remaining r different KV storage devices 330 in the parity group 340 (ie, k KV storages for which k key objects 350 are selected and erasure coded) Store K volume storage devices from the queue containing the device 330). Therefore, each of the k key objects 350 and the r parity objects 460 is stored in different ones of the N storage devices 330, and the corresponding data is stored in the N KV storage devices 330 of the parity group 340. Distributed evenly over each of the

  For packing reliability mechanisms using traditional erasure coding, reading and writing are relatively simple, but parity recalculation and restoration is not as simple. For parity recovery and recalculation (eg, in the case of updates), to know which key objects 350 are grouped together in the same parity group 340, thereby enabling parity calculations Information about the group of key objects 350 includes the actual value size 462 of each value object 450 (that is, the value size 462 excluding the virtual zero padding 464) and the KV storage device 330 (for example, the KV storage device 130 of FIG. 1). May be stored as metadata objects. Thus, in the current embodiment, additional metadata is provided for the original of each of the key objects 350 (eg, key objects 350 located in the trust group 140 of FIG. 1) and the value objects 450 corresponding to the key objects 350. It may be used to store the KV storage device 130 in length and key object 350 coding order.

  For example, the metadata object value includes a field indicating the entire key object 350 of the credibility group 140 and a value size 462 of the value object 450, and a parity object key (ie, virtual zero padding 464). A value object 450 including zero, a value size 462 of the parity object 460, and different fields indicating device IDs for identifying the corresponding r KV storage devices 330 in which the r parity objects 460 are stored. obtain.

  Data is stored using the user key. The metadata is stored in an internal key formed using the user key and a MetaID indicator called "Metadata" (display, indicator). Further, by determining where the value object 450 is terminated and where the zero of the virtual zero padding 464 begins, the value size 462 is meta-metabolized for accurate reconstruction when the value object 450 is regenerated. Stored in the data to allow recognition of the location of the virtual zero padding 464 (ie where the zero was added).

  If one of the KV storage devices 330 fails, all of the data and metadata may be stored in the same KV storage device 330, so there is a possibility that the data and metadata may be lost. Yes, which can make recovery impossible. However, in order to prevent such a situation, the metadata object value is a virtual object management layer 120 that can implement the reliability mechanism of the object replication previously mentioned for the metadata object value. It can be replicated using the "Object Replication Engine".

  In addition, since the metadata object value is the same for all objects in the reliability group 140, when the KV storage device 330 supports object linking, the same metadata object value is the same. Multiple key names that are commonly located in the KV storage device 330 can be linked. Furthermore, if batch writing is supported, object values can be batched together for better processing power.

To summarize the packing reliability mechanism using traditional erasure coding according to the current embodiment, the virtual device management layer 120 is stored in the k most recently stored k different KV storage devices 330 from the k different KV storage devices 330. The key object 350 can be selected.
Thereafter, the virtual device management layer 120 collects the value objects 450 (excluding the maximum size value object 470 of the parity group 440) corresponding to each key object 350, and performs padding with the virtual zero padding 350 to obtain the value object. 450 may be generated to the same size (e.g., the size of the largest sized value object 470).
The virtual device management layer 120 can subsequently generate r parity objects from the k key objects 350 using the MDS code process.
The virtual device management layer 120 thereafter sets r parity objects 460 to r KV storage devices different from the k KV storage devices 330 for which the key object 350 is selected among the N KV storage devices 330. It can write to 330. At this time, k + r is the same as N.
Thereafter, the virtual device management layer 120 can generate a metadata object indicating the above-described information.
Finally, the virtual device management layer 120 can write the key object 350 and the parity object 460 to N KV storage devices 330 (e.g., similar to replication engine) with a key formed from the user key and the metadata identifier. .

  FIG. 5 illustrates a single object (k, r) erasure coding or traditional “splitting” confidence using traditional erasure coding according to an embodiment of the present invention. FIG. 6 is a block diagram illustrating a group of KV storage devices configured to store key-value data in accordance with a gender mechanism.

Referring to FIG. 5, for a value having a value size larger than the value size compatible with the packing reliability mechanism using the object replication and traditional erasure coding mentioned earlier, the virtual device The management layer 120 may select a "Single-Object (k, r) erasure coding" or "splitting" reliability mechanism using traditional erasure coding.
The reliability mechanism of splitting using traditional erasure coding has a relatively large value size, and when divided into k equal-sized value objects 550, it has KV value (key-value with good processing capability) It is a reliability mechanism in units of KV values (key-value data / KV pairs) conforming to data / KV pairs 570.

  After dividing the KV value 570, according to the embodiment, the virtual device management layer 120 calculates a checksum for each of the k value objects 550. Thereafter, the virtual device management layer 120 inserts metadata before each of the k value objects 550.

  Splitting using traditional erasure coding divides the KV value 570 into a plurality of smaller value objects 550 and then divides the KV value 570 into k consecutive values objects 550. Distributed over the KV storage devices 530. Accordingly, the size of the k identically sized value objects 550 can be supported by each of the underlying KV storage devices 530.

  When splitting using traditional erasure coding is used, the virtual device management layer 120 is a systematic MDS code (for example, the virtual device layer 120 is a basic (de_fault) code, (4, 2) Reed Solomon code, etc. The r parity values / objects 560 generated using the traditional (k, r) MDS erasure coding scenario can also be added. Thereafter, in a manner similar to the packing reliability mechanism using the traditional erasure coding described above, the virtual device management layer 120 converts the k value objects 550 and the r parity objects 560 into N KV storages. The device 530 can be written (k + r = N).

  Thus, the virtual device management layer 120 can divide the relatively large KV value 570 into k value objects 550, calculate and add r parity objects 560, and can add k value objects 550 and r parity objects. 560 can be stored in (k + r) KV storage devices 530.

When using a splitting reliability mechanism using traditional erasure coding, after hashing the key 580 corresponding to the KV value 570 value, the virtual device management layer 120 stores the corresponding value object (eg, k In order to store the first value object among the number of value objects 550, D1 in the embodiment of FIG. 5 at the hash mark zero, the main KV storage device 530a (eg, in the embodiment illustrated in FIG. KV-SSD 2) can be determined. The k + r value objects 550, 560 may be written to each of the main KV storage device 530a and the (N-1) consecutive KV storage devices 530 under the same user key name.
That is, in the embodiment illustrated in FIG. 5, the first value object among the k value objects 550 is written in the main KV storage device 530a “KV-SSD2”, And the rest are written to the KV storage devices 530 “KV-SSD3” to “KV-SSDN” and “KV-SSD1” in the order of the circulation system together with the r parity objects 560 (for example, the reliability of the object duplication described above) A scheme similar to that described for the mechanism).

  To summarize the splitting reliability mechanism using traditional erasure coding, the virtual device management layer 120 can divide a relatively large KV value 570 into k identically sized value objects 550. The virtual device management layer 120 may subsequently generate r parity objects 560 for k value objects 550 using an MDS code process. Thereafter, the virtual device management layer 120 may hash the key corresponding to the value of the KV value 570 to determine the main KV storage device 530 a where the first value object should be located. The virtual device management layer 120 is subsequently generated by the virtual device management layer 120, and the appropriate MetaID fields corresponding to the main KV storage device 530a and the (N-1) consecutive KV storage devices 530 in the order of the circulation method are Under the same user key name, (k + r) value objects 550 and parity objects 560 can be written.

  Referring again to FIGS. 3 and 4, according to another embodiment, the virtual device management layer 120 uses K-object (k, r, d) erasure coding, ie, multi-object (K), using regeneration and erasure coding. A multiple_object “packing” reliability mechanism can be selected (eg, according to the sequence diagram 200 of FIG. 2), in which the virtual device management layer 120 packs k objects into k KV storage devices. And is similar to the reliability mechanism of packing using traditional erase coding as previously described, but packing using regenerative erase coding is an alternative to using traditional (k, r) erase codes Use the (k, r, d) regenerated code, so for the present embodiment, FIGS. It may be referred to as a common embodiment.

Therefore, if the regeneration code is appropriate, but it is not appropriate to divide the object, or if it is more appropriate to maintain the object without dividing it, the regeneration erasure coding is performed. The packing used may be used. Packing using regeneration and erasure coding is appropriate for the object replication previously mentioned, as well as value sizes larger than the value size used for reliability mechanisms of splitting and packing using traditional erasure coding. possible.
Packing using regenerative erasure coding may be used when reading multiple subpackets of an object does not result in lower performance than reading of the entire object. Packing using regeneration and erasure coding can be assisted during recovery / reconfiguration by the underlying KV storage device (eg, KV storage device 130 of FIG. 1 or KV storage device 330 of FIG. 3). It may be appropriate if it is a KV storage device that is aware of the generated code.

  FIG. 6 illustrates the reliability mechanism of “Single_Object (k, r, d) erasure coding”, ie “splitting” using regeneration coding according to an embodiment of the present invention. 2 is a block diagram showing a group of KV storage devices configured to store key value data.

  Referring to FIG. 6, the current reliability mechanism is a virtual device, except that it only uses (k, r, d) regeneration code instead of traditional (k, r) MDS erasure coding. The management layer may allow for operation in a manner similar to splitting using traditional erasure coding, as illustrated in FIG. Similar to packing using regenerative erasure coding, the current reliability mechanism is a KV storage device in which the underlying KV storage device 630 can assist during recovery / reconfiguration and can recognize the regenerated code. In some cases it may be appropriate.

  KV value object 670 has a larger value size than the KV value object corresponding to the reliability mechanism described previously, and reads a plurality of subpackets 690 of k splits (value object) 650 of KV value object 670 Splitting using regenerative erasure coding if does not result in lower performance than reading the entire split (value object) 650 (eg, performed with a splitting reliability mechanism using traditional erasure coding) May be appropriate.

  A splitting reliability mechanism that uses regenerative erasure coding is to split a KV value object 670 into k equal-sized value objects (splits) 650, each split 650 having multiple subpackets 690 (eg, In the current embodiment, the split 650 is virtually split into four subpackets 690), and the reading of a plurality of subpackets 690 from the KV value object 670 is a KV value object. It is a KV value object (KV pair) unit mechanism that can be adapted to a KV value object having a very large value size that can have an appropriate throughput if it has better processing capacity than 670 total reads. . Value size is supported by all underlying KV storage devices 630.

  Similar to splitting using traditional erasure coding, as shown in FIG. 5, the virtual device management layer 120 of the current reliability mechanism uses the systematic regeneration code to reconstruct r parity objects 660. Additionally, k splits (value objects) 650 and r parity objects 660 may be written to N KV storage devices 630 (where (k + r) = N). However, each of the r parity objects 660 may be divided into a plurality of parity subpackets 692 (the number corresponding to the number of subpackets 690 per split (value object) 650, for example). Unlike splitting, which uses traditional erasure coding, in the present embodiment, the default code may be a (4, 2, 5) zigzag code.

To summarize the reliability mechanism of splitting using regeneration and erasure coding, the virtual device layer 120 can divide a large KV value object 670 into k equal-sized splits (value objects) 650. Thereafter, the virtual device management layer 120 can divide each of the k objects 650 into m sub-packets 690 having the same size (k and m are integers). Virtual device management layer 120 subsequently generates r parity objects 660 for k objects 650 using a regenerative coding process, each of r parity objects 660 being m identical in size. Can be divided into parity subpackets 692 of
Thereafter, the virtual device management layer 120 can determine the main KV storage device 630a (in this case, KV-SSD_2) where the first split (value object) D1 is located by hashing the key corresponding to the KV value object 670. The virtual device management layer 120 is subsequently generated by the virtual device management layer 120, and the primary KV storage device 630a (in this case, KV-SSD_2) and (N-1) consecutive KV storage devices 630 In this case, under the same user key name, which may include an appropriate MetaID field corresponding to KV-SSD_3 to KV-SSD_k + r, KV-SSD_1), k splits (each containing m subpackets 690). Value objects 650 (D1 to Dk) and r parity objects 660 (Dk + 1 to Dk + r) each including m parity subpackets 692 may be written.

As described above, the virtual device management layer can select an appropriate reliability mechanism from a group of reliability mechanisms for storage of key-value data based on one or more attributes of the key-value data. . Accordingly, the embodiments described herein contribute to performance improvements in the field of memory storage because each of the described reliability mechanisms can perform a single key recovery procedure.
When the entire memory device fails, the virtual device management layer according to the embodiment of the present invention can recover all the keys existing in the failed memory device and copy them to the new memory device. The virtual device management layer performs a duplicate operation on the entire key existing in the memory device adjacent to the failed memory device in the reliability group, and the key determined that the reliability mechanism is present in the failed memory device By performing key unit multiple operations on the key recovery and copy of the entire key can be achieved.

  A very large KV pair (= KV value) with a value size larger than the value size supported by a basic reliability mechanism (eg, according to the size constraint of the underlying storage device) is set by the reliability manager to multiple KV pairs. The embodiment described further contributes to an improvement in the memory storage field, since it is explicitly split up and the reliability mechanism stores multiple splits and split number information together with the metadata stored in the KV value.

  The embodiments described herein, even when specific terms are employed, are to be used and interpreted in a general and descriptive sense only and are not intended to be limiting. In some embodiments, features described in connection with a particular embodiment will be apparent to those skilled in the art to which the present invention pertains, unless explicitly stated otherwise in that particular embodiment. Attributes, and / or elements may be used in combination or independently with features, attributes, and / or elements described in connection with other embodiments. Accordingly, those skilled in the art will recognize that various modifications in form or detailed description may depart from the spirit and scope of the present disclosure as expressed in the following claims, along with the functional equivalents contained in the present disclosure. It will be understood that it can be embodied without.

110 single virtual device 120 virtual device management layer 130 KV storage device 140 reliability group 170 key value data, key value data / KV pair 330, 330-1, ..., 330-N KV storage device 340 parity group / erase Code group 350, 350-i, 350-j, 350-y, 350-b, 350-k, 350-c Key / key object 450 Value object 460 Parity object 462 User value length / value size 464 “0” Filling / virtual zero / virtual zero padding 470 Maximum size value object 530 Storage device 530a Main KV storage device 550 Value object 560 Parity object 570 KV value / key value data / KV pair 580 key 630 KV strike Over its own device 630a major KV storage device 650 split (value object)
660 parity object 670 KV value object 690 subpacket 692 parity subpacket

Claims (20)

In a data storage method of a key-value reliability system including N (where N is an integer) storage devices grouped into one reliability group as a single logical unit and managed by a virtual device management layer,
Determining whether the data meets a threshold corresponding to a reliability mechanism for storing the data;
Selecting the reliability mechanism if the data is satisfied with the threshold value;
Storing the data according to the selected reliability mechanism.   The threshold may be an object size of the data, a throughput consideration of the data, a read / write temperature of the data (i.e., a read / write frequency), and the N storage devices. The method according to claim 1, characterized in that it is based on one or more of the underlying erasure coding capabilities of.   The method of claim 1, further comprising testing the data for the reliability mechanism using one or more bloom filters or caches.   The selected reliability mechanism, one or more checksums for each of the N storage devices storing the data, and the data stored in each of the N storage devices storing the data. The metadata for recording the size of the value object and the position of the parity group member of the N storage devices indicating where the data is stored in the N storage devices. The method of claim 1, further comprising: inserting with a corresponding key. The selected reliability mechanism includes object replication;
The step of storing the data comprises
Selecting a KV value,
Calculating a hash for hashing a key corresponding to the selected KV value;
Determining a subset of storage devices among the N storage devices to store a replica of a key object corresponding to the KV value;
Writing an updated value corresponding to the KV value into each of the determined storage device subsets under the same user key name. The method described in.
The selected reliability mechanism includes packing;
The step of storing the data comprises
Selecting k key objects stored in k (where k is an integer) storage devices among the N storage devices of the grouped reliability group;
Collecting k value objects corresponding to the k key objects;
Padding virtual zeros at the end of value objects that do not have the largest value size among the k value objects, so that the virtual values of all the k value objects are the same;
Generating r (where r is an integer) parity objects from the k key objects;
Writing the k key objects to the k storage devices;
Writing the r parity objects to r storage devices among the N storage devices;
The method of claim 1, wherein each of the r storage devices is distinguished from the k storage devices, where k + r = N. The selected reliability mechanism includes packing using traditional erasure coding
7. The method of claim 6, wherein the N storage devices are configured with a traditional (k, r) MDS (maximum distance separate) erasure coding. The selected reliability mechanism includes packing using regenerative erasure coding;
The method according to claim 6, wherein the N storage devices are configured with (k, r, d) regeneration and erasure coding.
The selected reliability mechanism includes splitting;
The step of storing the data comprises
Selecting a KV value,
Dividing the KV value into k (where k is an integer) equal sized value objects;
Generating r (where r is an integer) parity objects from the k equal-sized value objects;
Calculating a hash for hashing a key corresponding to the selected KV value;
Determining a main storage device in which the KV value is located among the N storage devices based on the hash;
Writing each of the k value objects of the same size and the r parity objects to the N storage devices in a sequential order starting from the main storage device, where k + r = N A method according to claim 1, characterized in that. The selected reliability mechanism includes splitting using traditional erasure coding;
The method of claim 9, wherein the N storage devices are configured with a traditional (k, r) maximum distance separable (MDS) erasure coding. The selected reliability mechanism includes packing using regenerative erasure coding,
The N storage devices are configured by (k, r, d) regeneration erasure coding,
The step of storing the data comprises
Dividing each of the k equal-sized value objects into m (where m is an integer) subpackets using the regenerative erasure coding;
The method of claim 9, further comprising: dividing each of the r parity objects into m parity subpackets.
In a data reliability system that stores data based on a selected reliability mechanism,
N (where N is an integer) storage devices configured as virtual devices using stateless data protection;
A virtual device configured to manage the N storage devices as the virtual device according to the selected reliability mechanism and store data in a storage device selected from the N storage devices. Device management layer, and
The virtual device management layer is
Determining whether the data satisfies a threshold corresponding to a reliability mechanism for storing the data;
If the data satisfies the threshold, select the reliability mechanism;
A data reliability system, characterized in that it is configured to store the data according to the selected reliability mechanism. The selected reliability mechanism includes object replication;
The virtual device management layer is
Select KV Value,
Calculating a hash for hashing a key corresponding to the selected KV value;
Determining a subset of storage devices among the N storage devices to store a replica of the key object corresponding to the KV value;
Configured to store the data by writing an updated value corresponding to the KV value into each of the determined storage device subsets under the same user key name, The data reliability system according to claim 12, characterized in that: The selected reliability mechanism includes packing
The virtual device management layer is
Select k key objects stored in k (where k is an integer) storage devices among the N storage devices,
Collect k value objects corresponding to the k key objects,
Padding virtual zeros at the end of value objects that do not have the largest value size among the k value objects, so that all virtual value sizes of the k value objects are the same,
R (where r is an integer) parity objects are generated from the k key objects,
Writing the k key objects to the k storage devices;
The r data is stored by writing the r parity objects to r storage devices among the N storage devices,
The data reliability system of claim 12, wherein each of the r storage devices is distinguished from each of the k storage devices, where k + r = N. The selected reliability mechanism includes splitting;
The virtual device management layer is
Select KV Value,
Divide the KV value into k (where k is an integer) equal sized value objects,
Create r (where r is an integer) parity objects from the k equal-sized value objects,
Calculating a hash for hashing a key corresponding to the selected KV value;
Determine a main storage device in which the KV value is located among the N storage devices based on the hash;
The data is stored by writing each of the k value objects of the same size and the r parity objects to the N storage devices in a sequential order starting from the main storage device. ,
13. The data reliability system according to claim 12, wherein k + r = N. The selected reliability mechanism includes splitting using regenerative erasure coding;
The N storage devices are configured with (k, r, d) regeneration and erasure coding,
The virtual device management layer divides the k value objects of the same size into m (where m is an integer) subpackets using the regenerative erasure coding, and each of the r parity objects. 16. The data reliability system of claim 15, further configured to store the data by dividing the data into m parity subpackets.
Method of storing data in a key-value reliability system comprising N storage devices grouped into one reliability group as a single logical unit and managed by a virtual device management layer when executed on a processor In a non-transitory computer readable medium containing computer code embodying
The method
Determining whether the data meets a threshold corresponding to a reliability mechanism for storing the data;
Selecting the reliability mechanism if the data is satisfied with the threshold value;
Storing the data in accordance with the selected reliability mechanism. The selected reliability mechanism includes object replication;
The step of storing the data comprises
Selecting a KV value,
Calculating a hash for hashing a key corresponding to the selected KV value;
Determining a subset of storage devices among the N storage devices to store a replica of a key object corresponding to the KV value;
Writing the updated value corresponding to the KV value into each of the determined storage device subsets under the same user key name. Non-transitory computer readable medium according to claim 1. The selected reliability mechanism includes packing
The step of storing the data comprises
Selecting k key objects stored in k (where k is an integer) storage devices among the N storage devices of the reliability group;
Collecting k value objects corresponding to the k key objects;
Padding virtual zeros at the end of the value objects not having the largest value size among the k value objects so that the virtual value sizes of all the k value objects are the same;
Generating r (where r is an integer) parity objects from the k key objects;
Writing the k key objects to the k storage devices;
Writing the r parity objects to r storage devices among the N storage devices;
The non-transitory computer-readable medium of claim 17, wherein each of the r storage devices is distinct from the k storage devices, and k + r = N. The selected reliability mechanism includes splitting;
The step of storing the data comprises
Selecting a KV value,
Dividing the KV value into k (where k is an integer) equal sized value objects;
Generating r (where r is an integer) parity objects from the k equal-sized value objects;
Calculating a hash for hashing a key corresponding to the selected KV value;
Selecting a main storage apparatus in which the KV value is located among the N storage apparatuses based on the hash;
Writing each of the k same size value objects and the r parity objects to the N storage devices in sequential order starting from the main storage device;
The non-transitory computer readable medium according to claim 17, characterized in that k + r = N.

Images