JP7171452B2 - Key-value data trust system, data storage method thereof, and non-transitory computer-readable medium containing computer code embodying the method - Google Patents

Description

The present disclosure (hereinafter also simply referred to as the present invention) relates generally to data storage systems, and more particularly to highly reliable key-value data storage in a key-value trusted system including multiple key-value storage devices. method for selecting a reliability mechanism that allows storage of

Data reliability mechanisms such as erasure coding are used to resolve data loss due to storage device failures and 'data corruptions' in a variety of installations containing multiple storage devices. can be used for

Conventional solid state drives (SSDs) generally use only a block interface and employ erasure coding or replication through a RAID (redundant array of independent disks) method. can be used to provide data reliability.
As object formats become diverse in size and unstructured, efficient data conversion is required at the interface between object and block levels. Furthermore, there is a need to ensure data reliability while maintaining space efficiency and fast access time characteristics.

Techniques such as RAID have been extensively investigated for conventional block storage devices. However, newer key-value storage devices have different interfaces and different storage semantics than traditional block devices. Therefore, for a wide variety of new key-value storage devices, there is the potential for key-value data and new data reliability techniques bespoke or specially adapted for key-value storage devices. have a nature.

U.S. Pat. No. 9,594,633 U.S. Pat. No. 9,639,268

The object of the present invention is to provide data for a key-value reliability system containing N (where N is an integer) storage devices grouped into one reliability group as a single logical unit and managed by a virtual device management layer. To provide a storage method.

Embodiments of the present invention each perform a single key recovery procedure whose trust mechanism allows the virtual device management layer to recover all keys residing on the failed memory device and copy them to a new memory device. As such, the embodiments described herein can contribute to improvements in the memory storage field.

(1) According to an embodiment of the present invention, it contains N (where N is an integer) storage devices grouped into one reliability group as a single logical unit and managed by the virtual device management layer. A data storage method for a key-value trust system is provided.
the method comprising determining whether the data satisfies a threshold corresponding to a reliability mechanism for storing the data; selecting the reliability mechanism if the data satisfies the threshold; and storing the data according to the selected reliability mechanism.

(2) The threshold is the object size of the data, throughput consideration of the data, read/write temperature (i.e., read/write frequency) of the data, and the N Based on one or more of the underlying erasure coding capabilities of the storage device.
(3) The method further includes testing the data against the reliability mechanism using one or more bloom filters or caches.
(4) the selected reliability mechanism, one or more checksums for each of the N storage devices storing the data, a checksum stored on each of the N storage devices storing the data; metadata for recording the size of the value object of the data, and the location of the parity group members of the N storage devices indicating where the data is stored among the N storage devices; The step of inserting with a key corresponding to the data is further included.
(5) the selected reliability mechanism includes object replication, and storing the data includes selecting a KV value; and calculating a hash for hashing a key corresponding to the selected KV value. , determining a subset of storage devices among the N storage devices for storing a replica of the key object corresponding to the KV value; and under the same user key name, determining the determined writing an updated value corresponding to the KV value to each in a subset of storage devices.

(6) The selected reliability mechanism includes packing, and the step of storing the data includes k among the N storage devices of the grouped reliability group, where k is selecting k key objects stored in (integer) storage devices; retrieving k value objects corresponding to the k key objects; padding with virtual zeros the ends of value objects that do not have the largest value size among the k value objects so that they are identical, r (where r is an integer) from the k key objects; writing the k key objects to the k storage devices; and writing the r parity objects to r storage devices among the N storage devices. each of said r storage devices is distinct from said k storage devices, where k+r=N.

(7) the selected reliability mechanism includes packing using traditional erasure coding, and the N storage devices are traditional (k, r) MDS (maximum distance separable) erasure coding; Configured.
(8) The selected reliability mechanism includes packing using regenerative erasure coding, and the N storage devices are configured with (k, r, d) regenerative erasure coding.

(9) the selected reliability mechanism includes splitting, and the storing of the data includes selecting a KV value, dividing the KV value into k (where k is an integer) objects of the same size; generating r (where r is an integer) parity objects from the k same-sized objects; calculating a hash for hashing a key corresponding to the selected KV value; determining a primary storage device in which the KV value is located among the N storage devices based on a hash; to the N storage devices, where k+r=N.

(10) The selected reliability mechanism includes splitting using traditional erasure coding, and the N storage devices are configured with traditional (k, r) MDS (maximum distance separable) erasure coding. .
(11) the selected reliability mechanism includes packing using regenerative erasure coding, wherein the N storage devices are configured with (k, r, d) regenerative erasure coding to store the data; divides the k same-sized value objects into m (where m is an integer) subpackets using the regeneration erasure coding; and dividing each of the r parity objects into m Further comprising dividing into parity subpackets.

(12) According to another embodiment of the present invention, there is provided a data reliability system that stores data based on a selected reliability mechanism. N storages, where N is an integer, configured as virtual devices using stateless data protection in the data reliability system, wherein the data reliability system stores data based on a selected reliability mechanism. According to the device and the selected reliability mechanism, the N storage devices are managed as the virtual devices and data is stored in a storage device selected from the N storage devices. wherein the virtual device management layer determines whether the data satisfies a threshold corresponding to a reliability mechanism for storing the data, and the threshold is met by the data. If so, it is configured to select the reliability mechanism and store the data according to the selected reliability mechanism.

(13) the selected reliability mechanism includes object replication, the virtual device management layer selects a KV value, calculates a hash for hashing a key corresponding to the selected KV value, determining a subset of storage devices among the N storage devices for storing key object replicas corresponding to values, and storing data in the determined subset of storage devices under the same user key name; is configured to store the data by writing an updated value corresponding to the KV value to each of the .
(14) The selected reliability mechanism includes packing, and the virtual device management layer stores k keys stored on k (where k is an integer) storage devices among the N storage devices. select an object, retrieve k value objects corresponding to the k key objects, and perform a calculation among the k value objects such that all the k value objects have the same virtual value size. pad the end of value objects that do not have the largest value size with virtual zeros, generate r (where r is an integer) parity objects from the k key objects, convert the k key objects to the k storage devices, and writing the r parity objects to r storage devices among the N storage devices, wherein the data is stored in the r storage devices; is distinguished from each of the k storage devices, where k+r=N.

(15) the selected reliability mechanism includes splitting, wherein the virtual device management layer selects a KV value and divides the KV value into k (where k is an integer) value objects of equal size; generating r parity objects from the k same-sized value objects, where r is an integer, calculating a hash for hashing a key corresponding to the selected KV value, and based on the hash determining a main storage device in which the KV value is located among the N storage devices, and each of the k same-sized value objects and the r parity objects in consecutive order starting from the main storage device; to the N storage devices, where k+r=N.
(16) the selected reliability mechanism includes splitting using regeneration erasure coding, the N storage devices are configured with (k, r, d) regeneration erasure coding, and the virtual device management layer comprises: dividing the k same-sized objects into m (where m is an integer) subpackets using the regeneration-erasure coding, and dividing each of the r parity objects into m parity subpackets; By partitioning, it is further configured to store said data.

(17) According to another embodiment of the present invention, when executed on a processor, N storages grouped into one reliability group as a single logical unit and managed by the virtual device management layer. A non-transitory computer-readable medium is provided that includes computer code embodying a method for storing data in a key-value trust system that includes an apparatus. the method comprising determining whether the data satisfies a threshold corresponding to a reliability mechanism for storing the data; selecting the reliability mechanism if the data satisfies the threshold; and storing data according to the selected reliability mechanism.

(18) The selected reliability mechanism includes object replication, and storing the data includes selecting a KV value and calculating a hash for hashing a key corresponding to the selected KV value. , selecting a subset of storage devices among the N storage devices to store a replica of the key object corresponding to the KV value; and storing the KV value under the same user key name. writing corresponding updated values to each in the determined subset of storage devices.
(19) the selected reliability mechanism includes packing, and storing the data in k (where k is an integer) storage devices among the N storage devices of the reliability group; selecting the k key objects obtained; collecting k value objects corresponding to the k key objects; padding with virtual zeros the ends of value objects that do not have the largest value size among the value objects; generating r (where r is an integer) parity objects from the k key objects; writing the k key objects to the k storage devices; and writing the r parity objects to r storage devices among the N storage devices; are distinguished from the k storage devices, k+r=N.
(20) the selected reliability mechanism includes splitting, wherein storing the data comprises selecting a KV value, dividing the KV value into k (where k is an integer) value objects of equal size; generating r (where r is an integer) parity objects from the k same-sized value objects; and calculating a hash for hashing a key corresponding to the selected KV value. , selecting, based on the hash, a primary storage device among the N storage devices in which the KV value is located; and starting with the primary storage device and in consecutive order, the k same-sized values. writing an object and each of the r parity objects to the N storage devices, where k+r=N.

According to the reliability mechanism according to embodiments of the present invention, the virtual device management layer recovers all keys present in the failed memory device and each performs a single key recovery procedure that allows them to be copied to a new memory device. Therefore, the reliability of memory storage can be improved.

The ideas discussed above and/or other ideas will be further clarified and appreciated from the detailed description of the embodiments below in conjunction with the accompanying drawings.
1 is a block diagram illustrating a key-value trust system that stores key-value data based on selected trust mechanisms, in accordance with embodiments of the present invention; FIG. FIG. 4 is a flow diagram illustrating selection of a trust mechanism used by a key-value trust system based on a size threshold corresponding to the size of key-value pair data, according to an embodiment of the present invention; configured to store key-value data according to the reliability mechanism of K-object (k,r) erasure coding using traditional erasure coding, ie, multi-object “packing”, according to an embodiment of the present invention; 2 is a block diagram showing a group of KV storage devices arranged together; FIG. 4 is a block diagram illustrating storage of value objects and parity objects according to the reliability mechanism of K-object (k,r) erasure coding using traditional erasure coding, ie, multi-object “packing”, according to an embodiment of the present invention; FIG. is. Key-value data according to the trust mechanism of “Single_Object (k,r) erasure coding” or “Splitting” using traditional erasure coding according to an embodiment of the present invention. 2 is a block diagram illustrating a group of KV storage devices configured to store the . According to an embodiment of the present invention, key-values according to the reliability mechanism of “Single_Object (k, r, d) erasure coding” or “Splitting” using regenerative erasure coding. 1 is a block diagram illustrating a group of KV storage devices configured to store data; FIG.

The various features of the invention and the manner in which it is achieved can be understood in more detail with reference to the accompanying drawings and the following detailed description of the embodiments. In the following, embodiments are described in more detail with reference to the attached drawings. Like reference numbers refer to like elements throughout. This invention may, however, be embodied in various other forms and should not be construed as limited to the embodiments illustrated herein. Rather, such embodiments are provided as illustrations to aid in general understanding of the invention, and may fully convey the technical features and aspects of the invention to those skilled in the art. Accordingly, processes, elements and techniques that are unnecessary for those skilled in the art to fully appreciate the features and aspects of the present invention may not be described. Throughout the accompanying drawings and detailed description, like reference numerals refer to like elements and such descriptions are not repeated unless otherwise noted. In the drawings, the relative sizes of elements, layers, and regions may be exaggerated for clarity.

Various embodiments are described herein with reference to cross-sectional illustrations that are schematic illustrations of embodiments and/or intermediate structures. As such, variations from the illustrated configuration must be expected, for example, as a result of manufacturing techniques and/or tolerances. Moreover, any specific structural or functional descriptions herein are merely illustrations for describing embodiments consistent with the inventive concept. That is, the embodiments described herein are not to be construed as being limited to the specific illustrated configurations of the regions, and are to include deviations in shape that are, for example, the result of manufacturing. For example, implant regions illustrated as rectangles have traditionally been rounded at their edges or have curvilinear features and/or gradients of implant concentration, rather than a binary transition from implant to non-implant region. can be done. Similarly, a buried region formed by an implant can cause some implant in the region between the buried region and the surface from which the implant occurs. That is, the areas illustrated in the drawings are schematic in nature and their shapes are not intended to illustrate the actual shape of the area of the device and are not limiting thereof. Additionally, those skilled in the art will recognize that the described embodiments can be modified in various other ways without departing from the spirit or scope of the invention.

In the following detailed description, a number of specific statements are provided for convenience of explanation and to aid in understanding the various embodiments. However, various embodiments can be implemented without detailed description or with one or more equivalent alternatives. In other instances, well-known structures and devices are shown in block diagram form in order not to obscure the various embodiments unnecessarily.

Terms such as “first,” “second,” “third,” etc. may be used in text to describe various elements, structures, regions, hierarchies, and/or areas. Although used, it should be understood that such elements, configurations, regions, hierarchies, and/or areas are not limited to such terms. Such terms are only used to distinguish one element, structure, region, hierarchy or section from another element, structure, region, hierarchy or section. Thus, a first element, configuration, region, hierarchy or section described below could be termed a second element, configuration, region, hierarchy or section without departing from the spirit and scope of the present invention. .

Spatially relative terms such as "beneath, below, lower, under", "above, upper", etc. are defined in terms of other terms illustrated in the drawings. May be used in text to readily describe the relationship of one element or feature to another. It will be appreciated that spatially relative terms are intended to include other orientations of the device in operation or use in addition to those illustrated in the drawings. For example, if the device is turned over in a drawing, an element described as "below or beneath or under" another element or feature would be "above" the other element or feature. . That is, the exemplary terms "below, under" can include both directions above and below. The device may be oriented in other directions (eg, rotated 90° or in other directions) and the spatially relative statements used in the text should be interpreted accordingly. Similarly, when we say that a first portion is aligned "on" a second portion, this means that the first portion is positioned on top of the second portion, with no limitation to its top surface based on the direction of gravity. Indicates aligned to the face or bottom face.

When an element, hierarchy, region, or structure is referred to as being “on, connected to, or coupled to” another element, hierarchy, region, or structure, it refers to the other element, hierarchy, region, or structure. Or, there may be one or more intermediate elements, hierarchies, regions, or configurations that are directly linked to the configuration. However, the term "directly connected" refers to one component being directly connected to another component with no intermediate structure. On the other hand, other expressions describing relationships between constructs such as "between, immediately between" or "adjacent to or directly adjacent to" can be similarly interpreted. , when an element or hierarchy is referred to as being between two elements or hierarchies, there may be only one element or hierarchy between the elements or structures, or there may be one or more intermediate elements or hierarchies. It can be appreciated that there can be more hierarchies.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, singular terms are intended to include plural forms unless the context clearly dictates otherwise. When the term “comprising” is used in the detailed description, it defines the presence of the recited features, integers, steps, acts, elements and/or configurations, but one or more of the other features, integers , does not exclude the presence or addition of steps, acts, elements, structures, and/or groups thereof. As used herein, the term "and/or" includes any combination or subset of one or more of the associated listed listings.

As used herein, the terms “substantial,” “about, approximately,” and similar terms are used as terms of approximation and terms of degree. Rather, it is intended to describe deviations inherent in the measured or calculated values that can be recognized by one of ordinary skill in the art. The term "about, approximately" as used herein includes the value referred to and takes into account the error associated with the measurement of the specified quantity and the measurement in question (e.g., limitations of the measurement system). It means within an acceptable deviation from the specified value as determined by one skilled in the art. For example, "about" can mean within one or more standard deviations, or within ±30%, 20%, 10%, 5% of the stated value. can. As used herein, the terms "use, using, and used" can be considered synonymous with "utilizing, utilizing, and utilized." Also, the term "exemplary" is intended to refer to an "example or illustration."

When a particular embodiment is implemented differently, the particular process order may be performed differently than the order described. For example, two sequentially described processes can be performed substantially simultaneously or in the reverse order of the described order.

Electrical or electronic devices and/or other associated devices or configurations according to embodiments of the invention described herein may be implemented using suitable hardware, firmware (eg, application-specific integrated circuits (ASICs)). ), software, or a combination of software, firmware, and hardware, for example, various configurations of such devices may be implemented in one integrated circuit (IC) chip or in another In addition, various configurations of such devices can be formed on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB). circuit board, or formed on a single substrate.In addition, various configurations of such devices can be used in other systems to perform various functions described herein. One or more processor-driven processes or threads in one or more computing devices that communicate with a configuration and execute computer program instructions, which are stored in random access memory (RAM). The computer program instructions are stored in a memory that can be embodied in a computing device using standard memory devices such as CD-ROMs, flash drives, etc. It will be apparent to those skilled in the art that the functionality of various computing devices may be incorporated into a single computing device without departing from the spirit and aspects of the exemplary embodiments of the present invention. It can be understood that the functionality of a particular computing device can be combined or integrated, or that the functionality of a particular computing device is distributed among one or more other computing devices.

Unless defined otherwise, all terms including technical/scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in common dictionaries are to be construed to have a meaning consistent with their meaning in the context of the art to which they relate and/or in this detailed description of the invention and are not explicitly defined in the text. Unless explicitly defined, it should not be interpreted in an ideal or too formal sense.

As described below, embodiments of the present disclosure provide a key-value trust system (KV) consisting of multiple key-value (KV) storage devices grouped into a single logical unit. It provides a way to reliably store key-value data in a reliability system. Additionally, embodiments of the present invention provide a stateless hybrid reliability manager that manages drives and controls storage of key-value (KV) pairs. The stateless hybrid trust manager implements Object Replication; K-Object (k, r) erasure coding-Packing; Single Object (k, r) Single Object (k, r) erasure coding-Splitting; K-Object (k, r, d) regeneration coding-Packing; Relies on multiple pluggable reliability mechanisms/techniques/implementations including Single Object (k,r,d) regeneration coding-splitting .

A stateless hybrid trust manager that relies on multiple pluggable trust mechanisms can manage the device and control the storage of KV pairs, and the methods disclosed for trust mechanism selection are of different size KV pairs. The described embodiments can improve the performance of memory storage (eg, storing key-value data in key-value storage devices) by ensuring efficient storage, retrieval, and retrieval.

FIG. 1 is a block diagram illustrating a key-value trust system that stores key-value data based on a selected trust mechanism, according to an embodiment of the invention.

Referring to FIG. 1, as described above, for a variety of new key-value (KV) "storage devices" {aka "memory devices", "drives", "KV-SSDs"} (130), key-value data and There is an implicit possibility that new data reliability mechanisms that are tailored or specially adapted for the KV storage device 130 will come in handy. Accordingly, such a hybrid key-value reliability system for the KV storage device 130 uses the hybrid reliability mechanism in response to one or more pluggable reliability mechanisms. , and controls the storage of KV pairs in them.
Although solid-state drives (SSDs) are commonly used to refer to KV storage devices described herein, other storage devices may be used in accordance with embodiments of the present invention. The design and operation of the virtual device management layer 120 according to embodiments of the present invention are described below.

In the current embodiment, the virtual device management layer 120 can enable a method of reliably storing key-value data/KV pairs 170 in a key-value trust system. The key-value trust system includes multiple KV storage devices 130 grouped into a single logical unit. A single logical unit is called a reliability group 140 .

KV storage device 130 of trust group 140 stores each chunk of erasure-coded and/or replicated data corresponding to key-value data 170 . A KV storage device 130 in trust group 140 is exposed as a single virtual device 110 ostensibly for which key-value operations are directed to through virtual device management layer 120 .

Virtual device 110 includes a stateless hybrid reliability manager as virtual device management layer 120 . That is, the virtual device management layer 120 operates in a stateless manner (ie, does not require maintaining a mapping between key-values and devices).

Therefore, the virtual device 110 stores key-value data through N KV storage devices 130 (where N is an integer) (eg, KV-SSDs 130-1, 130-2, 130-3, 130-4, . . . , 130-N). It stores data 170 and stores key-value data 170 in the KV storage device 130 through the virtual device management layer 120 . That is, the virtual device management layer 120 manages the KV storage device 130 and controls the storage of KV pairs therein.

In another embodiment, the key-value trust system may further include a cache that selectively stores metadata and/or data associated with the key of the key-value data 170 to improve operation speed. A trust mechanism can attach the value of a KV pair to include metadata corresponding to the KV pair. That is, both the key and value can be appended with information corresponding to the metadata identifier "MetaID" to store additional metadata specific to that KV pair. The metadata includes a checksum, a reliability mechanism identifier for identifying the reliability mechanism used to store the data, an erasure code identifier, object size, parity group number location, and the like.

In other embodiments, the key-value trust system may further include bloom filters corresponding to the trust mechanisms described below. Bloom filters can store stored keys using corresponding trust mechanisms, thereby supporting key-value trust systems in read operations. Thus, one or more bloom filters or caches of a key-value trust system can enable rapid testing of keys against existing trust mechanisms.

Each of the trust mechanisms described herein uses the same hash function on the corresponding single-digit key modulo in the KV storage device 130 to hash the first of the KV pair corresponding to the key-value data 170 . can be stored first. That is, for each pluggable trust mechanism, the trust mechanism can store at least the first copy/chunk using the same key as the user key.

As described above, embodiments of the present invention provide multiple pluggable reliability mechanisms that ensure reliable storage of key-value data 170 on multiple KV storage devices 130 . Therefore, the virtual device management layer 120 needs a reliability mechanism and can decide which of the reliability mechanisms to use.

Reliability mechanisms may be based on policies such as value-size thresholds set during virtual device 110 setup and/or object read/write frequency. Accordingly, the virtual device management layer 120 can select the appropriate reliability mechanism based on the system's specified policy.

How the reliability mechanisms are operated by the virtual device management layer 120 and when the reliability mechanisms are appropriately used and selected for the five reliability mechanisms of embodiments of the present invention are described below. describe. The five reliability mechanisms are Object Replication, K-Object (k, r) erasure coding-packing, and Single Object (k, r) erasure. Coding-Splitting (Single Object (k, r) erasure coding-Splitting), K-Object (k, r, d) regeneration coding-Packing (K-Object (k, r, d) regeneration coding-Packing), and single It can be referred to as Single Object (k, r, d) regeneration coding-splitting.

FIG. 2 is a flow diagram (200) illustrating the selection of trust mechanisms used by a key-value trust system based on a size threshold corresponding to the size of data in a KV pair, according to an embodiment of the present invention.

Referring to FIG. 2, for all supported reliability mechanisms based on size thresholds (eg, the five reliability mechanisms described above), the virtual device management layer 120 uses data (eg, key-value data 170). ), determine whether the value-size is less than a given threshold (t _i ) corresponding to each reliability mechanism, and determine the first reliability mechanism that satisfies the value-size threshold requirements. You can choose.

For example, at S210, the virtual device management layer 120 receives "n" (where n is an integer) reliability mechanisms supported on a size threshold basis. At S220, the virtual device management layer 120 simply considers each of the reliability mechanisms, one at a time, in order from 1 to n. At S230, upon consideration of each reliability mechanism to be supported, the virtual device management layer 120 determines whether the value size of the data is less than the threshold _ti corresponding to each reliability mechanism.

At S240, if a reliability mechanism is found with a threshold t _i that is greater than or equal to the value size of the data, the virtual device management layer 120 chooses to use that reliability mechanism. At S250, either the reliability mechanism used at S240 has been determined, or no reliability mechanism among the N reliability mechanisms has an appropriate threshold t _i that satisfies the value size at the final iteration of S220. If so, virtual device management layer 120 is finished determining the reliability mechanism to be used.

In the current embodiment, "n" equals 5 depending on the five different reliability mechanisms of the embodiments described herein. For relatively very small key-values (ie, when the value size is relatively small), the virtual device management layer 120 chooses to use the Object Replication reliability mechanism. For slightly larger key-values, the virtual device management layer 120 selects the reliability mechanism of packing and then Splitting (e.g., in order) while using traditional erasure coding for each. do. However, for larger key-values, the virtual device management layer 120 chooses packing and then splitting, while using regeneration erasure coding instead of traditional erasure coding. .

In embodiments of the present invention, the selection of the reliability mechanism for use depends on the object size of the object, the processing power requirements for the object, the read/write temperature of the corresponding key-value pairs, i.e. read/write temperature. It may be based on one or more of write frequency, basic 8underlying coding capabilities of KV storage devices, and/or detection of whether a key is hot or cold. For example, "hot" keys use the reliability mechanism of Object Replication regardless of their value size, while "cold" keys depend on their value size. It is applied to one of the reliability mechanisms of the erasure coding scheme. As another embodiment, the decision whether to use the Object Replication reliability mechanism may be based on both size and read/write temperature. Accordingly, a threshold corresponding to object read/write temperature, ie object read/write frequency, can be used to determine the reliability mechanism instead of the threshold corresponding to size in flowchart 200 of FIG.

The operation of each of the five reliability mechanisms is described below.

Referring back to FIG. 1, as previously mentioned, when the value size of a KV pair is relatively small, the “Object Replication” reliability mechanism is the appropriate choice by the virtual device management layer 120. be. Object Replication is applied per object (eg, per key-value data/KV pair 170). The Object Replication reliability mechanism has high storage overhead, but low read and restore costs, making it suitable for very small value sizes.

Object Replication reliability mechanisms are also appropriate for key-values (eg, key-value data 170) that have frequent updates, and are therefore selected based on read and write frequency.

During Object Replication, the key-value data 170 is replicated to one or more additional KV storage devices 130 each time a write occurs. A primary copy of the key-value data 170 is located in one of the KV storage devices 130 designated by a key modulo (N) hash. Replicas of the primary copy of the key-value data 170 are placed in sequential or immediately adjacent KV storage devices 130 in a circular fashion.

Virtual device management layer 120 or the user decides how many copies of key-value data 170 to create. For example, a distributed system using the virtual device management layer 120 may select 3-way replication and set 3-way replication as the default. However, the user of the system can configure the number of object replicas to be more or less than the selected default.

Thus, for example, if three-way replication is used and the primary KV storage device 130-2 contains the primary copy of data (eg, key-value data 170), the virtual device management layer 120 will replicate the primary copy of the data. are stored in subsequent replicated KV storage devices 130-3, 130-4, and all copies of the data are identical. That is, a copy of the data may be transferred to two (or more) directly following KV storage devices 130-3, 130-4 (eg, in a circular fashion) following the KV storage device 130-2 containing the primary copy of the data. stored in

A copy of the data is stored in the duplicate KV storage devices 130-3, 130-4 under the same keyname/same user key as the primary KV storage device 130-2. All copies of data contain an identifier and checksum to identify the replicated key-value data 170 .

Therefore, if a specific KV storage device 130 is faulty (for example, the KV storage device 130-3 is faulty), the KV storage devices 130 directly before and after the faulty KV storage device 130-3 ( For example, by recovering the value using a recovery mechanism for the key name to the KV storage devices 130-2, 130-4 directly before and after the KV storage device 130-3, the duplicated key is restored.

To summarize the reliability mechanism of Object Replication, the virtual device management layer 120 receives the key-value data 170, hashes the key object, and stores the replica of the key object. 130 is determined. The virtual device management layer 120 will subsequently pass the updated value to the selected KV storage device 130 (eg, the selected KV storage device 130) under the same user key name (eg, the appropriate MetaID field). -2, 130-3, 130-4).

FIG. 3 illustrates K-Object (k, r) erasure coding using traditional erasure coding (K-Object (k, r) erasure coding) or multiple objects according to an embodiment of the present invention. 1 is a block diagram illustrating a group of KV storage devices configured to store key-value data according to a multiple object “Packing” reliability mechanism; FIG.

Referring to FIG. 3, the reliability mechanism of packing using traditional erasure coding is that small value Selected for data with size. For example, the reliability mechanism of packing using traditional erasure coding has a larger value size (but still relatively is selected by the virtual device management layer 120 for data with

Packing using traditional erasure coding consists of traditional (k, r) MDS (maximum distance separable) erasure coding and is used with a systematic MDS code. As an example, the erasure code is basically a (by_default), (4,2) Reed-Solomon Code, because the (4,2) Reed-Solomon Code has been relatively widely studied. , and corresponding high-speed implementation libraries are readily available.

N different KV storage devices 330-1, . k keys/key objects 350 from queues of k different KV storage devices 330 that are part of are selected, erasure coded and packed (where k is an integer).

For example, virtual device management layer 120 maintains a buffer of recently written key objects 350 for each KV storage device 330 (eg, each KV storage device 130 in trust group 140 of FIG. The device management layer 120 selects k key objects 350 from k different KV storage devices to enable erasure coding, thereby packing k key objects 350 corresponding to KV pairs.

In the current embodiment, the virtual device management layer 120 selects four key objects 350x, 350y, 350b, 350c respectively from four different KV storage devices 330-1, 330-3, 330-4, 330-N ( In the current embodiment, k=4).

FIG. 4 illustrates K-object (k,r) erasure coding using traditional erasure coding, or storage of value objects and parity objects according to the reliability mechanism of multi-object “packing”, according to an embodiment of the present invention. It is a block diagram showing.

3 and 4, the key object 350 is located in the 'hash of key modulo n' th KV storage device 330 . That is, each hash of key modulo n is performed for each key object 350 and transmitted to the queue of that particular KV storage device 330 . In the current embodiment, the i-th key (Key _i ) 350-i is hashed and located on KV-SSD1 (330-1) and the j-th key (Key _j ) 350-j is hashed and located on KV-SSD2 ( 330-2), the kth key (Key _k ) 350-k is hashed and located in KV-SSD4 (330-4).

The user value length/value size 462 of each stored value object 450 is the same as the written user value length/value size. However, for consistency to enable erasure coding, user value/value objects 450 are all the same size by having "0" filling/virtual zero/virtual zero padding 464 attached to them. is considered to have
That is, since the user value size 462 of each different value object 450 can vary (i.e., the value object 450 has a variable length or is a variable length key-value), the virtual zero By implementing the padding 464 method (i.e., padding the value object 450 with the zeros of the virtual zero padding 464 for coding purposes, while actually rewriting the data representing the value object 450 including the padded zeros). ), the parity object 460 has the same size as the largest size value object 470 in the parity group 340 .
Therefore, in the current embodiment, value objects 450 "Val x", "Val y", and "Val b" are padded with virtual zeros that are not actually stored in the KV storage device, thereby maximizing the size of value object 470. It is assumed to have the same size as "Val c". Therefore, a parity object 460 is computed.

After coding the k key objects 350 , the virtual device management layer 120 computes r parity objects 460 from the k value/k value objects 450 corresponding to the k key objects 350 . Then r is an integer and k+r=N, where N is the number of KV storage devices 330 in parity group 340 (eg, N KV storage devices 130 in reliability group 140 of FIG. 1).

The virtual device management layer 120 assigns the r parity objects 460 to the remaining r different KV storage devices 330 in the parity group 340 (that is, k key objects 350 are selected and erasure-coded k KV storages). r KV storage devices are partitioned from the queue containing device 330). Therefore, each of the k key objects 350 and the r parity objects 460 is stored in a different one of the N storage devices 330, and the corresponding data is stored in the N KV storage devices 330 of the parity group 340. are evenly distributed in each of the

For packing reliability mechanisms that use traditional erasure coding, reading and writing are relatively straightforward, but recalculating and recovering parity is less straightforward. For parity restoration and recomputation (e.g., for updates), to know which key objects 350 are grouped together in the same parity group 340, thereby enabling parity computation: Information about the group of key objects 350 is stored in the KV storage device 330 (eg, KV storage device 130 of FIG. 1) along with the actual value size 462 of each value object 450 (i.e., the value size 462 minus the virtual zero padding 464). can be stored as a metadata object in each of the . Therefore, in the current embodiment, additional metadata is provided for each of the key objects 350 (eg, the key objects 350 located in the trust group 140 of FIG. 1) and the value objects 450 corresponding to the key objects 350. The length and coding order of the key object 350 can be used to store the KV storage device 130 .

For example, the metadata object value indicates the entirety of the key object 350 of the reliability group 140 and includes a field that indicates the value size 462 of the value object 450, and the parity object key (i.e., virtual zero padding 464). Value object 450 including zero), value size 462 of parity object 460, and different fields indicating device IDs for identifying corresponding r KV storage devices 330 in which r parity objects 460 are stored. obtain.

Data is stored using a user key. Metadata is stored in an internal key formed using the user key and a MetaID indicator called "Metadata". Additionally, by determining where the value object 450 ends and where the zeros of the virtual zero padding 464 begin, the value size 462 is metabolized for accurate reconstruction when the value object 450 is regenerated. It is stored in the data to allow recognition of the location of the virtual zero padding 464 (ie, where the zeros were added).

If one of the KV storage devices 330 fails, all of the data and metadata may be stored in the same KV storage device 330, so there is a potential loss of data and metadata. Yes, which may make recovery impossible. However, in order to prevent such a situation, the metadata object value is defined as a "virtual device management layer 120" that can implement the previously mentioned Object Replication reliability mechanism for the metadata object value. It can be replicated using the "Object Replication Engine".

Additionally, since the metadata object value is the same for all objects in the reliability group 140, the same metadata object value is the same if the KV storage device 330 supports object linking. It can be linked to multiple keynames commonly located in the KV storage device 330 . Furthermore, if batch writing is supported, object values can be batched together for better throughput.

To summarize the packing reliability mechanism using traditional erasure coding according to the current embodiment, the virtual device management layer 120 stores the k most recently stored data from k different KV storage devices 330 via buffers. A key object 350 can be selected.
The virtual device management layer 120 then collects the value objects 450 corresponding to each key object 350 (excluding the maximum size value object 470 of the parity group 440), and pads them with virtual zero padding 350 to obtain the value objects. 450 can be generated to have the same size (eg, the size of the largest value object 470).
The virtual device management layer 120 can then generate r parity objects from the k key objects 350 using the MDS code process.
The virtual device management layer 120 then divides the r parity objects 460 into r KV storage devices different from the k KV storage devices 330 in which the key object 350 is selected among the N KV storage devices 330 . 330 can be written. At this time, k+r is the same as N.
The virtual device management layer 120 can then generate a metadata object indicating the above information.
Finally, the virtual device management layer 120 can write the key object 350 and parity object 460 along with the key formed from the user key and metadata identifier to the N KV storage devices 330 (eg, analogous to replication engine). .

FIG. 5 shows Single Object (k, r) erasure coding using traditional erasure coding, or the reliance on "splitting", according to an embodiment of the present invention. 1 is a block diagram illustrating a group of KV storage devices configured to store key-value data according to a security mechanism; FIG.

Referring to FIG. 5, for a value with a value size greater than the value size that conforms to the previously mentioned object replication and packing reliability mechanisms using traditional erasure coding, a virtual device The management layer 120 can select a reliability mechanism of "Single_Object (k,r) erasure coding", or "splitting", which uses traditional erasure coding.
A splitting reliability mechanism that uses traditional erasure coding has a relatively large value size and a KV value (key-value data/KV pair) 570 is a trust mechanism per KV value (key-value data/KV pair).

After partitioning the KV value 570, the virtual device management layer 120 computes a checksum for each of the k value objects 550, according to an embodiment. Virtual device management layer 120 then inserts metadata in front of each of the k value objects 550 .

Splitting, using traditional erasure coding, divides the KV value 570 into multiple smaller value objects 550, and then divides the multiple smaller value objects 550 obtained by splitting the KV value 570 into k consecutive distributed across multiple KV storage devices 530; Accordingly, the size of k same-sized Value Objects 550 can be supported by each of the underlying KV storage devices 530 .

When using splitting with traditional erasure coding, the virtual device management layer 120 uses a systematic MDS code (e.g., the virtual device layer 120 uses a (4, 2) Reed-Solomon code, etc. as the base (de_fault) code). r parity values/objects 560 generated using (k, r) MDS erasure coding) can also be added. Thereafter, in a manner similar to the reliability mechanism of packing using traditional erasure coding described above, virtual device management layer 120 stores k value objects 550 and r parity objects 560 in N KV storages. Device 530 can be written (k+r=N).

Therefore, the virtual device management layer 120 can divide the relatively large KV value 570 into k value objects 550, calculate and add r parity objects 560, and calculate k value objects 550 and r parity objects. 560 can be stored in (k+r) KV storage devices 530 .

When using a splitting reliability mechanism that uses traditional erasure coding, after hashing the key 580 corresponding to the value of the KV value 570, the virtual device management layer 120 stores the corresponding value object (e.g., k To store the first value object among the value objects 550, D1 in the embodiment of FIG. KV-SSD2) can be determined. The k+r Value Objects 550, 560 can be written to each of the primary KV storage device 530a and the (N-1) consecutive KV storage devices 530 under the same user key name.
That is, in the embodiment shown in FIG. 5, the first value object among the k value objects 550 is written to the main KV storage device 530a “KV-SSD2”, and the k value objects 550 are , the rest are written to the KV storage devices 530 "KV-SSD3" to "KV-SSDN" and "KV-SSD1" along with the r parity objects 560 in circular order (for example, the object replication reliability described above). in a manner similar to that described for the mechanism).

To summarize the reliability mechanism of splitting using traditional erasure coding, the virtual device management layer 120 can split a relatively large KV value 570 into k equal-sized value objects 550 . The virtual device management layer 120 can then generate r parity objects 560 for the k value objects 550 using the MDS code process. The virtual device management layer 120 can then hash the key corresponding to the value of the KV value 570 to determine the primary KV storage device 530a where the first value object should be located. The virtual device management layer 120 then populates the appropriate MetaID fields corresponding to the primary KV storage device 530a and the (N−1) consecutive KV storage devices 530 generated by the virtual device management layer 120 in circular order. (k+r) value objects 550 and parity objects 560 can be written under the same user key name.

3 and 4, according to another embodiment, the virtual device management layer 120 uses regenerative erasure coding, K-object (k, r, d) erasure coding, ie, multiple object ( A reliability mechanism of multiple_object "packing" can be selected (eg, according to sequence diagram 200 of Figure 2).The current reliability mechanism is that virtual device management layer 120 packs k objects into k KV storage devices. , similar to the reliability mechanism of packing using traditional erasure coding described previously, but packing using regenerative erasure coding instead of using the traditional (k,r) erasure code , we use the (k, r, d) regenerated code, so Figures 3 and 4 can be referred to as generic embodiments for the current embodiment.

Therefore, if regeneration code is appropriate, but it is not appropriate to split the object, or even better to intact the object without splitting or otherwise, then regeneration erasure coding is appropriate. Any packing used may be used. Packing using regeneration erasure coding is appropriate for value sizes larger than those used for the previously mentioned object replication and reliability mechanisms of splitting and packing using traditional erasure coding. could be.
Packing using regenerative erasure coding may be used when reading multiple subpackets of an object does not result in lower performance than reading the entire object. Packing using regenerative erasure coding allows the underlying KV storage device (e.g., KV storage device 130 of FIG. 1 or KV storage device 330 of FIG. 3) to assist during restoration/reconstruction and regenerate. It may be suitable if the KV storage device is aware of the synthetic code.

FIG. 6 illustrates a reliability mechanism of “Single_Object (k, r, d) erasure coding” using regenerative erasure coding, ie, “Splitting”, according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a group of KV storage devices configured to store key-value data on a network;

Referring to FIG. 6, the current reliability mechanism uses (k,r,d) regenerate code instead of traditional (k,r) MDS erasure coding. A management layer may enable operation in a manner similar to splitting using traditional erasure coding, as illustrated in FIG. Similar to packing using regenerated erasure coding, the current reliability mechanism is a KV storage device that the underlying KV storage device 630 can assist during recovery/reconfiguration and recognize the regenerated code. may be appropriate in some cases.

The KV-value object 670 has a larger value size than the KV-value object corresponding to the previously described reliability mechanism, and the reading of multiple subpackets 690 of k splits (value objects) 650 of the KV-value object 670. does not result in lower performance than reading the entire split (value object) 650 (e.g., accomplished with the reliability mechanism of splitting using traditional erasure coding). may be appropriate.

A reliability mechanism for splitting using regeneration-erasure coding is that a KV value object 670 is split into k equal-sized value objects (splits) 650, each of which splits 650 into multiple subpackets 690 (e.g., In the current embodiment, the split 650 is virtually further split into four subpackets 690 each, and the reading of multiple subpackets 690 from the KV Value Object 670 is performed by the KV Value Object. A per-KV-value-object (KV-pair) mechanism that can accommodate KV-value objects with a value size that is so large that it can have reasonable throughput if it has better processing power than the entire 670 read. . Valuesize is supported by all underlying KV storage devices 630 .

Similar to splitting using traditional erasure coding, as illustrated in FIG. Additionally, k splits (value objects) 650 and r parity objects 660 can be written to N KV storage devices 630 (where (k+r)=N). However, each of the r parity objects 660 may be split into a plurality of parity subpackets 692 (the number of which corresponds, for example, to the number of subpackets 690 per split (value object) 650). Unlike splitting using traditional erasure coding, in the current embodiment the default code may be a (4,2,5) zigzag code.

To summarize the splitting reliability mechanism using regenerate-erasure coding, the virtual device layer 120 can split a large KV value object 670 into k equal-sized splits (value objects) 650 . The virtual device management layer 120 can then divide each of the k objects 650 into m equal-sized subpackets 690 (where k and m are integers). The virtual device management layer 120 then generates r parity objects 660 for the k objects 650 using a regeneration coding process, each of the r parity objects 660 having m identical size of parity subpackets 692 .
The virtual device management layer 120 can then hash the key corresponding to the KV value object 670 to determine the primary KV storage device 630a (KV-SSD_2 in this case) where the first split (value object) D1 is located. The virtual device management layer 120 is subsequently created by the virtual device management layer 120 and consists of the primary KV storage device 630a (in this case KV-SSD_2) and the (N-1) consecutive KV storage devices 630 (in this case, KV-SSD_2) in a circular fashion. In this case, k splits each containing m subpackets 690 ( value objects) 650 (D1 through Dk), and r parity objects 660 (Dk+1 through Dk+r) each containing m parity subpackets 692 may be written.

According to what has been described above, the virtual device management layer can select an appropriate reliability mechanism from a group of reliability mechanisms for storing key-value data based on one or more attributes of the key-value data. . Accordingly, the embodiments described herein contribute to performance improvements in the field of memory storage, as each of the described trust mechanisms can perform single-key recovery procedures.
In the event of an entire memory device failure, the virtual device management layer of embodiments of the present invention can recover all of the keys residing on the failed memory device and copy them to the new memory device. The virtual device management layer performs an iterative operation over all of the keys present in memory devices adjacent to the failed memory device in the trust group, and the keys that the trust mechanism determines to be present in the failing memory device. Recovery and copying of the entire key can be achieved by performing per-key multiple operations on the .

If a very large KV pair (=KV value) with a value size larger than the value size supported by the underlying reliability mechanism (e.g., subject to size constraints of the underlying storage device) is determined by the reliability manager to handle multiple KV pairs , and the reliability mechanism stores multiple splits and split number information along with the metadata stored in the KV value, the described embodiments further contribute to improvements in the memory storage arena.

Although the embodiments described herein employ specific terminology, such terminology is to be used and construed in a generic and descriptive sense only and is not intended to be limiting. In some embodiments, as will be apparent to one of ordinary skill in the art to which this invention pertains, features described in connection with a particular embodiment unless explicitly stated to the contrary in that particular embodiment. , attributes and/or elements may be used in combination or independently with the features, attributes and/or elements described in connection with other embodiments. Thus, it will be appreciated by those skilled in the art that various changes in form or detailed description, along with functional equivalents contained in this disclosure, may depart from the spirit and scope of the present disclosure as expressed in the following claims. It will be appreciated that it may be embodied without

110 single virtual device 120 virtual device management layer 130 KV storage device 140 reliability group 170 key value data, key value data/KV pair 330, 330-1, ..., 330-N KV storage device 340 parity group/erase Code Groups 350, 350-i, 350-j, 350-y, 350-b, 350-k, 350-c Key/Key Object 450 Value Object 460 Parity Object 462 User Value Length/Value Size 464 “0” Filling/Virtual Zero/Virtual Zero Padding 470 Maximum Size Value Object 530 Storage Device 530a Primary KV Storage Device 550 Value Object 560 Parity Object 570 KV Value/Key Value Data/KV Pair 580 Key 630 KV Storage Device 630a Primary KV Storage Device 650 Split (value object)
660 parity object 670 KV value object 690 subpacket 692 parity subpacket

Claims (17)

A data storage method in a key-value reliability system comprising one or more storage devices grouped into a reliability group as a single logical unit and managed by a virtual device management layer, comprising:
determining, by the virtual device management layer, that the data satisfies a threshold corresponding to a reliability mechanism including object replication for storing the data;
using, by the virtual device management layer, the reliability mechanism identified by a reliability mechanism identifier contained in metadata that identifies the reliability mechanism;
storing , by the virtual device management layer, the data according to the reliability mechanism;
Storing the data by the virtual device management layer includes:
selecting a KV value;
calculating a hash for hashing a key corresponding to the KV value;
determining a subset of storage devices among the one or more storage devices for storing key object replicas corresponding to the KV value;
and writing an updated value corresponding to the KV value to a subset of the storage devices under the same user key name. The threshold includes object size of the data, throughput consideration of the data, read/write temperature of the data, and underlying erase coding of the one or more storage devices. 2. The method of claim 1, based on one or more of capabilities. 2. The method of claim 1, further comprising testing , by the virtual device management layer, the data against the reliability mechanism using a bloom filter or cache. The virtual device management layer controls the reliability mechanism, the metadata including checksums for the one or more storage devices storing the data, and the metadata stored on the one or more storage devices storing the data. said meta for recording the size of a value object of data and the location of parity group members of said one or more storage devices indicating where in said one or more storage devices said data is stored; 2. The method of claim 1, further comprising inserting data with a key corresponding to said data. the reliability mechanism includes packing;
Storing the data by the virtual device management layer includes:
selecting one or more key objects respectively stored in the same number of first respective one or more storage devices among the one or more storage devices of the reliability group;
retrieving one or more value objects corresponding to the one or more key objects;
padding the end of value objects that do not have the largest value size among the one or more value objects with virtual zeros so that the virtual value sizes of the one or more value objects are the same;
generating one or more parity objects from the one or more key objects;
writing the one or more key objects to the first respective one or more storage devices;
writing the one or more parity objects to the same number of second respective one or more storage devices among the one or more storage devices;
said second respective one or more storage devices are distinct from said first respective one or more storage devices;
The method of claim 1, wherein the sum of the number of the one or more key objects and the number of the one or more parity objects is the same as the number of the one or more storage devices. the reliability mechanism includes packing using erasure coding;
6. The method of claim 5, wherein the one or more storage devices are configured with maximum distance separable (MDS) erasure coding. the reliability mechanism includes packing using regenerative erasure coding;
6. The method of claim 5, wherein the one or more storage devices are configured with regeneration erasure coding. the reliability mechanism includes splitting;
Storing the data by the virtual device management layer includes:
dividing the KV value into one or more equal-sized objects;
generating one or more parity objects from the one or more same-sized objects;
determining a primary storage device where the KV value is located among the one or more storage devices based on the hash;
writing one or more Value Objects to the same number of first respective one or more storage devices among said one or more storage devices in consecutive order starting from said primary storage device; to a second respective one or more storage devices of the same number among the one or more storage devices;
said second respective one or more storage devices are distinct from said first respective one or more storage devices;
The method of claim 1, wherein the sum of the number of the one or more value objects and the number of the one or more parity objects is the same as the number of the one or more storage devices. the reliability mechanism includes splitting using erasure coding;
9. The method of claim 8, wherein the one or more storage devices are configured with maximum distance separable (MDS) erasure coding. the reliability mechanism includes splitting using regeneration erasure coding;
wherein the one or more storage devices are configured with regeneration erasure coding;
Storing the data by the virtual device management layer includes:
dividing the one or more same-sized objects into one or more subpackets using the regeneration erasure coding;
9. The method of claim 8, further comprising dividing the one or more parity objects into one or more parity subpackets. A data reliability system for storing data based on a reliability mechanism, comprising:
one or more storage devices configured as virtual devices using stateless data protection;
a virtual device management layer configured to manage the one or more storage devices as the virtual devices and store data in the one or more storage devices according to the reliability mechanism including object replication; prepared,
The virtual device management layer determines that the data satisfies a threshold corresponding to the reliability mechanism for storing the data, and includes a reliability mechanism identifier included in metadata to identify the reliability mechanism. configured to use the reliability mechanism identified by and store the data in response to the reliability mechanism;
The virtual device management layer selects a KV value, calculates a hash for hashing a key corresponding to the KV value, and stores a replica of the key object corresponding to the KV value. storing the data by determining a subset of storage devices among storage devices and writing an updated value corresponding to the KV value to the subset of storage devices under the same user key name; A data reliability system configured to: the reliability mechanism includes packing;
The virtual device management layer selects one or more key objects stored in the same number of first respective one or more storage devices among the one or more storage devices, and extracting one or more value objects corresponding to a key object , and a value object that does not have the largest value size among the one or more value objects so that the virtual value sizes of the one or more value objects are the same; to generate one or more parity objects from the one or more key objects , write the one or more key objects to the first respective one or more storage devices; configured to store the data by writing the one or more parity objects to the same number of second respective one or more storage devices among the one or more storage devices;
said second respective one or more storage devices are distinct from said first respective one or more storage devices;
12. The data reliability of claim 11, wherein the sum of the number of the one or more key objects and the number of the one or more parity objects is the same as the number of the one or more storage devices. sex system. the reliability mechanism includes splitting;
The virtual device management layer divides the KV value into one or more same-sized objects, generates one or more parity objects from the one or more same-sized objects, and generates the one or more parity objects based on the hash. determining a primary storage device in which said KV value is located among one or more storage devices, and placing one or more Value Objects into the same one among said one or more storage devices in sequential order starting with said primary storage device; write to a first respective one or more storage devices of the same number, and write one or more parity objects to a second respective one or more storage devices of the same number among the one or more storage devices configured to store the data by writing
said second respective one or more storage devices are distinct from said first respective one or more storage devices;
12. The data reliability of claim 11, wherein the sum of the number of the one or more value objects and the number of the one or more parity objects is the same as the number of the one or more storage devices. sex system. the reliability mechanism includes splitting using regeneration erasure coding;
wherein the one or more storage devices are configured with regeneration erasure coding;
The virtual device management layer divides the one or more same-sized objects into one or more subpackets using the regeneration erasure coding, and divides the one or more parity objects into one or more parity subpackets. 14. The data reliability system of claim 13, further configured to store the data by dividing it into packets. A method for storing data in a key-value reliability system comprising one or more storage devices grouped into a reliability group as a single logical unit and managed by a virtual device management layer when executed on a processor. A non-transitory computer-readable recording medium having computer code embodying:
The method includes:
determining, by the virtual device management layer, that the data satisfies a threshold corresponding to a reliability mechanism including object replication for storing the data;
using, by the virtual device management layer, the reliability mechanism identified by a reliability mechanism identifier contained in metadata that identifies the reliability mechanism;
storing , by the virtual device management layer, the data according to the reliability mechanism;
Storing the data by the virtual device management layer includes:
selecting a KV value;
calculating a hash for hashing a key corresponding to the KV value;
determining a subset of storage devices among the one or more storage devices for storing key object replicas corresponding to the KV value;
and writing an updated value corresponding to the KV value to a subset of the storage devices under the same user key name. the reliability mechanism includes packing;
Storing the data by the virtual device management layer includes:
selecting one or more key objects stored in the same number of first respective one or more storage devices among the one or more storage devices of the reliability group;
retrieving one or more value objects corresponding to the one or more key objects;
padding the end of value objects that do not have the largest value size among the one or more value objects with virtual zeros so that the virtual value sizes of the one or more value objects are the same;
generating one or more parity objects from the one or more key objects;
writing the one or more key objects to the first respective one or more storage devices;
writing the one or more parity objects to the same number of second respective one or more storage devices among the one or more storage devices;
said second respective one or more storage devices are distinct from said first respective one or more storage devices;
16. The non-temporary storage device of claim 15, wherein the sum of the number of the one or more key objects and the number of the one or more parity objects is the same as the number of the one or more storage devices. computer-readable recording medium. the reliability mechanism includes splitting;
Storing the data by the virtual device management layer includes:
dividing the KV value into one or more equal-sized objects;
generating one or more parity objects from the one or more same-sized objects;
determining a primary storage device where the KV value is located among the one or more storage devices based on the hash;
writing one or more Value Objects to the same number of first respective one or more storage devices among said one or more storage devices in sequential order starting from said primary storage device; to a second respective one or more storage devices of the same number among the one or more storage devices;
said second respective one or more storage devices are distinct from said first respective one or more storage devices;
16. The non-temporary storage device of claim 15, wherein the sum of the number of the one or more value objects and the number of the one or more parity objects is the same as the number of the one or more storage devices. computer-readable recording medium.

Images