CN111857602B - Data processing method, data processing device, data node and storage medium - Google Patents
Data processing method, data processing device, data node and storage medium Download PDFInfo
- Publication number
- CN111857602B CN111857602B CN202010758439.6A CN202010758439A CN111857602B CN 111857602 B CN111857602 B CN 111857602B CN 202010758439 A CN202010758439 A CN 202010758439A CN 111857602 B CN111857602 B CN 111857602B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- target storage
- snapshot
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of distributed storage, and provides a data processing method, a device, a data node and a storage medium, wherein the method comprises the following steps: receiving a data writing request sent by a client, wherein the data writing request carries an address to be written, and the address to be written is used for representing a first target storage block in which data to be written is to be written and a first target storage unit in the first target storage block; determining a target snapshot to be written in by data to be written according to the address to be written, wherein the target snapshot corresponds to a first target storage block, the target snapshot comprises a storage area, and the storage area corresponds to a storage unit in the first target storage block; writing data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to a first target storage unit; and returning a response message of successful writing to the client. The invention improves the writing performance of the distributed storage system by avoiding the reading and writing punishment generated when the data to be written is written into the storage block.
Description
Technical Field
The present invention relates to the field of distributed storage technologies, and in particular, to a data processing method, an apparatus, a data node, and a storage medium.
Background
In order to improve the reliability of data of the distributed storage system, the currently mainstream distributed storage system can well support block storage of the copy, and by storing the copy of the data, when the original data is damaged, the copy data can be continuously used, so that the service of the storage system is not influenced.
However, in the security field, because the amount of video data is large, the problem of low disk utilization rate cannot be solved by block storage in the conventional copy mode, data is stored based on erasure code coding, in order to ensure the reliability of distributed storage system data in the erasure code coding mode, a snapshot is usually created for a designated storage space, and when the data in the storage space is damaged, the data in the storage space is rolled back to a normal state during the snapshot through the snapshot.
When data is stored in a distributed storage system with a Copy-on-write (COW) snapshot, a written original data block is firstly copied to a specified position, and then the data to be stored is written into the original data block, so that one data writing operation needs to be completed by one reading operation (namely reading the original data block) and two writing operations (writing the original data block into the specified position and writing the data to be stored into the original data block), so that serious reading and writing penalties exist, and the writing performance of the distributed storage system is reduced.
Disclosure of Invention
The invention aims to provide a data processing method, a data processing device, a data node and a storage medium, which are used for creating a corresponding snapshot for a storage block and temporarily storing data to be written into the storage block into the snapshot corresponding to the storage block, so that the read-write punishment generated when the data to be written is written into the storage block is avoided, and the write performance of a distributed storage system is improved.
In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solutions:
in a first aspect, the present invention provides a data processing method, applied to a data node in a distributed storage system, where the data node includes multiple storage blocks, each storage block includes multiple storage units, the distributed storage system further includes a client, and the data node is communicatively connected to the client, and the method includes: receiving a data writing request sent by a client, wherein the data writing request carries an address to be written, and the address to be written is used for representing a first target storage block in which data to be written is to be written and a first target storage unit in the first target storage block; determining a target snapshot to be written in by data to be written according to the address to be written, wherein the target snapshot corresponds to a first target storage block, the target snapshot comprises a storage area, and the storage area corresponds to a storage unit in the first target storage block; writing data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to a first target storage unit; and returning a response message of successful writing to the client.
In a second aspect, the present invention provides a data processing apparatus applied to a data node in a distributed storage system, where the data node includes a plurality of storage blocks, each storage block includes a plurality of storage units, the distributed storage system further includes a client, and the data node is communicatively connected to the client, the apparatus includes: the data writing method comprises the steps that a receiving module is used for receiving a data writing request sent by a client, wherein the data writing request carries an address to be written, and the address to be written is used for representing a first target storage block in which data to be written is to be written and a first target storage unit in the first target storage block; the processing module is used for determining a target snapshot to be written in by data to be written according to the address to be written, wherein the target snapshot corresponds to the first target storage block, the target snapshot comprises a storage area, and the storage area corresponds to a storage unit in the first target storage block; writing data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to a first target storage unit; and the return module is used for returning a response message of successful writing to the client.
In a third aspect, the present invention provides a data node, comprising: one or more processors; a memory for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a data processing method as described above.
In a fourth aspect, the invention provides a computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, realizes the data processing method as described above.
Compared with the prior art, the method and the device have the advantages that the corresponding snapshot is created for the storage block, the data to be written in the storage block is temporarily stored in the snapshot corresponding to the storage block, the reading and writing punishment generated when the data to be written is written in the storage block is avoided, and the writing performance of the distributed storage system is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 shows a schematic view of an application scenario provided in an embodiment of the present invention.
Fig. 2 is a block diagram illustrating a data node according to an embodiment of the present invention.
Fig. 3 shows a flowchart of a data processing method according to an embodiment of the present invention.
Fig. 4 is a schematic diagram illustrating determining a target snapshot according to an embodiment of the present invention.
Fig. 5 is a flowchart illustrating another data processing method according to an embodiment of the present invention.
Fig. 6 is a flowchart illustrating another data processing method according to an embodiment of the present invention.
Fig. 7 is a flowchart illustrating another data processing method according to an embodiment of the present invention.
Fig. 8 is a flowchart illustrating another data processing method according to an embodiment of the present invention.
Fig. 9 is a flowchart illustrating another data processing method according to an embodiment of the present invention.
Fig. 10 is a flowchart illustrating another data processing method according to an embodiment of the present invention.
FIG. 11 shows a block schematic diagram of a data processing apparatus according to an embodiment of the present invention.
Icon: 10-a data node; 11-a processor; 12-a memory; 13-a bus; 14-a communication interface; 20-a client; 30-a management node; 100-a data processing device; 110-a receiving module; 120-a processing module; 130-return module.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined or explained in subsequent figures.
In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. indicate an orientation or a positional relationship based on that shown in the drawings or that the product of the present invention is used as it is, this is only for convenience of description and simplification of the description, and it does not indicate or imply that the device or the element referred to must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present invention.
Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.
Referring to fig. 1, fig. 1 shows a schematic view of an application scenario provided by an embodiment of the present invention, in fig. 1, a distributed storage system includes a data node 10, a client 20, and a management node 30, where the client 20 is communicatively connected to both the data node 10 and the management node 30, and the data node 10 is communicatively connected to the management node 30.
The data node 10 is used for storing data required to be stored by a user, and the data node 10 includes two storage areas: the first storage area includes a plurality of storage blocks, each storage block includes a plurality of storage units, the second storage area is used for storing a snapshot of each storage block, the snapshot of each storage block includes a storage area, the storage area corresponds to a storage unit in the storage block, when data in the storage unit of the storage block is consistent with data in the corresponding storage area of the snapshot, the storage area may include an address of the storage unit corresponding thereto, when there is data to be written in the storage unit of the storage block, the data to be written will be written in the storage area of the snapshot corresponding to the storage unit first, and at this time, the storage area may include the data to be written. The data node 10 periodically updates the data in the snapshot to the corresponding storage block. In a practical scenario, in order to further speed up the writing performance of the distributed storage system, the storage medium in the second storage area may employ a nonvolatile memory.
The client 20 receives data which is sent from the outside and needs to be stored by a user, segments the received data, and performs erasure coding on each segment of data to obtain a group of coded multi-piece data, wherein the number of pieces in the group of data is determined according to a preset erasure coding rule, for example, if an erasure coding rule of (5,3) is adopted, that is, each piece of original data is divided into 5 data blocks and 3 check blocks, the client 20 stores the 8 blocks onto different data nodes 10, and when any 3 blocks are in error (including the data blocks and the check blocks), the original 5 blocks can be restored through a corresponding reconstruction algorithm, thereby realizing redundant storage of the data and improving reliability of the data.
The management node 30 is used to store metadata of the management data node. For example, when the client 20 sends data to be written to a data node, it needs to first apply for the management node 30 for permission of a storage space for storing the data to be written, and the management node 30 returns a list of the data node 10 for writing the data to be written and a storage unit of a storage block in the data node 10 to the client 20.
The data node 10 may be a storage server for storing data, or a storage server group consisting of a plurality of storage servers.
The client 20 may be an entity computer such as one host or one server, a host group including a plurality of hosts, or a server group including a plurality of servers, or a virtual host or a virtual server, or a virtual host group or a virtual server group, which can realize the same function as the entity computer. The client 20 may be a stand-alone hardware or an application running on the data node 10 or the management node 30.
The management node 30 may be an entity computer such as a host or a server, a host group consisting of a plurality of hosts, or a server group consisting of a plurality of servers, or a virtual host or a virtual server, or a virtual host group or a virtual server group, which can implement the same function as the entity computer. The management node 30 may be a stand-alone hardware or an application running on a data node.
On the basis of fig. 1, an embodiment of the present invention further provides a block schematic diagram of the data node 10 in fig. 1, please refer to fig. 2, and fig. 2 shows a block schematic diagram of the data node 10 provided in the embodiment of the present invention. The data node 10 comprises a processor 11, a memory 12, a bus 13, a communication interface 14. The processor 11 and the memory 12 are connected by a bus 13, and the processor 11 is communicatively connected to the client 20 and the management node 30 by different communication interfaces 14.
The processor 11 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 11. The Processor 11 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components.
The memory 12 is used for storing a program, such as the data processing device, which includes at least one software functional module that can be stored in the memory 12 in a form of software or firmware (firmware), and the processor 11 executes the program after receiving an execution instruction to implement the data processing method disclosed in the above embodiment.
The Memory 12 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Alternatively, the memory 12 may be a storage device built in the processor 11, or may be a storage device independent of the processor 11.
The bus 13 may be an ISA bus, a PCI bus, an EISA bus, or the like. Fig. 2 is represented by only one double-headed arrow, but does not represent only one bus or one type of bus.
The communication connection between the data node 10 and the client 20 and the management node 30 is realized by at least one communication interface 14 (which may be wired or wireless).
On the basis of fig. 1 and fig. 2, an embodiment of the present invention provides a data processing method, which may be applied to the data node 10 in fig. 1 and fig. 2, please refer to fig. 3, and fig. 3 shows a flowchart of a data processing method provided by an embodiment of the present invention, where the method includes the following steps:
step S100, receiving a data writing request sent by a client, wherein the data writing request carries an address to be written, and the address to be written is used for representing a first target storage block in which data to be written is to be written and a first target storage unit in the first target storage block.
In this embodiment, the data node 10 includes a plurality of storage blocks, the size of the storage block may be set in advance as required, for example, the size of the storage block is set to 64MB, each storage block includes a plurality of storage units, a first target storage block is a storage block to which data to be written in the plurality of storage blocks should be written, a first target storage unit is a storage unit to which data to be written in the plurality of storage units in the first target storage block should be written, for example, the data node 10 includes 1000 storage blocks, storage block 1 to storage block 1000, each storage block includes 1000 storage units, storage unit 1 to storage unit 1000, and an address to be written is: if the storage block 700 and the storage unit 500 are the same, the storage block 700 is a first target storage block, and the storage unit 500 is a first target storage unit.
In this embodiment, when writing data to be written for the first time, in order to avoid that different clients 20 write data to be written to the same position of the same data node 10 and cause mutual overlapping of the data, the client 20 first applies a write permission with a preset space size to the management node before writing the data to the data node 10, where the preset space size may be a preset multiple of a storage unit, and when writing the data to the preset space, the client 20 uses a write request to carry an address to be written to indicate a first target storage unit in a first target storage block to which the data to be written needs to be written.
In this embodiment, when the client 20 segments and encodes the received data, if it is found that the position to which the encoded data needs to be written is not the start position of a storage unit, but is a certain position in the middle of the storage unit, at this time, the content of the storage unit needs to be read first, and then the data to be modified and encoded is written into the position corresponding to the storage unit, and then a data writing request for writing the storage unit is sent to the data node. For example, the size of the memory unit is 64KB, and the offset position where the encoded data needs to be written into the 10# memory unit is 60KB, the client 20 first temporarily stores the data read from the 10# memory unit, writes the encoded data into the 60KB position of the temporarily stored 10# memory unit, and then sends a write data request for writing into the 10# memory unit to the data node 10.
Step S110, determining a target snapshot to which data to be written is to be written according to the address to be written, where the target snapshot corresponds to the first target storage block, and the target snapshot includes a storage area, and the storage area corresponds to a storage unit in the first target storage block.
In this embodiment, before writing the data to be written into the target snapshot, where each storage area is the same as the data of the corresponding storage unit, at this time, an address of the storage unit corresponding to the storage area may be stored in the storage area, so that the data in the storage area may be found according to the address of the storage unit, and after writing the data to be written into the storage area of the target snapshot, the storage area includes the written data to be written or the actual address where the data to be written is written, in addition to the address of the storage unit corresponding to the storage area, so that the written data to be written may be found.
Step S120, writing the data to be written into a first target storage area in the storage areas, where the first target storage area corresponds to the first target storage unit.
The first target storage area is a storage area corresponding to the first target storage unit among the storage areas, for example, the 1# storage area corresponds to the 1# storage unit, the first target storage unit is the 1# storage unit, and the 1# storage area is not the first target storage area.
Step S130, a response message of successful writing is returned to the client.
According to the data processing method provided by the embodiment of the invention, the corresponding snapshot is created for the storage block, the data to be written in the storage block is temporarily stored into the target snapshot corresponding to the storage block, and the read-write punishment generated when the data to be written is written in the storage block is avoided, so that the write performance of the distributed storage system is improved.
On the basis of fig. 2, an embodiment of the present invention provides a specific implementation manner of determining a target snapshot, please refer to fig. 4, fig. 4 shows a flowchart of another data processing method provided in the embodiment of the present invention, and step S110 includes the following sub-steps:
in sub-step S1101, if there is no snapshot in the first target storage block or there is one snapshot in the first target storage block and data in the first target storage area corresponding to the first target storage unit in the snapshot has been updated, a snapshot of the first target storage block is newly created, and the newly created snapshot is taken as a target snapshot.
In this embodiment, based on the feature that the security monitoring field only needs to ensure the correctness of data rollback once, the requirement of the feature can be satisfied when at most two snapshots exist in each storage block.
In this embodiment, it is first determined whether a snapshot exists in the first target storage block, and if the snapshot does not exist in the first target storage block, a snapshot is created for the first target storage block, and the created snapshot is used as a target snapshot; if the first target storage block only has one snapshot, judging whether data in a first target storage area corresponding to the first target storage unit in the snapshot is updated or not, if so, creating the snapshot for the first target storage block, taking the created snapshot as the target snapshot, and if not, not needing to create a new snapshot, and taking the snapshot of the first target storage block as the target snapshot.
In the substep S1102, if there is a snapshot in the first target storage block and the data in the first target storage area corresponding to the first target storage unit in the snapshot is not updated, the snapshot is taken as the target snapshot.
In sub-step S1103, if there are two snapshots in the first target storage block, the target snapshot is determined from the two snapshots according to a preset rule.
In this embodiment, each snapshot corresponds to a first target storage block, each snapshot includes a first target storage area corresponding to a first target storage unit, when data in the first target storage area is updated, a version number and a state corresponding to the first target storage area are updated, the state is a to-be-confirmed state or a confirmed state, after a group of data is subjected to erasure coding, the client 20 sends the group of data to a plurality of data nodes 10 for storage, when the client 20 receives a response that the storage is successful by feeding back a preset number of data nodes 10, the group of data is successfully stored, at this time, a confirmation request is sent to the data node 10 that the storage is successful in feedback, before the data node 10 receives the confirmation request sent by the client 20, the data stored this time is in an unconfirmed state, and after the confirmation request sent by the client 20 is received, the data stored this time is updated from the unconfirmed state to the confirmed state.
As a specific implementation manner, the method for determining the target snapshot according to the preset rule may be:
firstly, if the data stored in the first target storage area in any snapshot is not updated, the snapshot corresponding to the first target storage area of which the data is not updated is taken as the target snapshot.
And secondly, if the data stored in the first target storage area in the two snapshots are not updated, taking any one snapshot in the two snapshots as a target snapshot.
And thirdly, if the data stored in the first target storage area in the two snapshots are updated and the state of the first target storage area with a large version number is a confirmation state, taking the snapshot corresponding to the first target storage area with a small version number as the target snapshot.
In this embodiment, the data in the first target storage area with the larger version number is the latest written data, the state of the first target storage area with the larger version number is the confirmed state, and the data in the first target storage area corresponding to the version number is represented to be correctly stored.
To facilitate description of the several cases of determining the target snapshot, the embodiments of the present invention provide schematic diagrams of the several cases of determining the target snapshot, please refer to fig. 5, fig. 5 shows a schematic diagram of determining the target snapshot provided by the embodiments of the present invention, and fig. 5 (a), fig. 5 (b), and fig. 5 (c) respectively correspond to the three cases.
According to the data processing method provided by the embodiment of the invention, by utilizing the characteristic that the security monitoring field only needs to ensure the correctness of data rollback once, two snapshots are created for each storage block at most, the data to be written is firstly written into the unmodified target snapshot or the data is written into the target snapshot of the first storage area with a smaller version number after the latest data is confirmed to be successfully written, so that on the basis of meeting the data reliability, the storage space occupied by the snapshot data is not excessive, the waste of the excessive snapshot data to the storage space is reduced, and the utilization rate of the storage space is improved.
On the basis of fig. 2, in order to manage the state of data to be written, and ensure that the data can be written into the first target storage unit only after being actually written successfully, before a confirmation request sent by a client to confirm successful writing of the data is not received, an embodiment of the present invention further provides a specific implementation manner for ensuring data reliability of a target storage area, please refer to fig. 6, where fig. 6 shows a flowchart of another data processing method provided by the embodiment of the present invention, and the method further includes the following steps:
step S140, increment the version number of the first target storage area and set the state of the first target storage area to the to-be-confirmed state.
In this embodiment, each first target storage area corresponds to one version number, and when the data in each first target storage area is updated, the corresponding version number is incremented, so as to ensure that the data to be written in the same first target storage unit, which needs to be written in different writing time, can be distinguished. In this embodiment, when there are two snapshots and the data in the first target storage area in any one snapshot is updated, the corresponding version number is incremented, for example, the version number of the 1# storage area of the 1# snapshot is 1, when data is written to the 1# storage area of the 2# snapshot, the version number of the 1# storage area of the 2# snapshot is incremented on the basis of the version number of 1 and is 2, when data is written to the 1# storage area of the 1# snapshot, the version number of the 1# storage area of the 1# snapshot is 3, and if there is an update in the data in the 1# storage area of the 1# snapshot at this time, the version number of the 1# storage area of the 1# snapshot is 4.
In this embodiment, the to-be-confirmed state is used to characterize a previous state in which data belonging to the same group of erasure coding codes as data in the corresponding memory region has not yet been determined to be successfully stored.
In the data processing method provided by the embodiment of the invention, when the data in the first target storage area is updated, the version number is increased progressively to ensure that the data to be written in the same storage unit, which need to be written in at different writing time, can be distinguished, and before the data of the same group of erasure correcting codes are not determined to be successfully stored, the state of the storage area storing the group of data is set to be an unconfirmed state, so that the reliable recording of the data state in the storage area is ensured.
In order to update the state of the data in time when it is determined that the data is successfully stored, an embodiment of the present invention provides a specific implementation manner for updating the state of the first target storage area, please refer to fig. 7, and fig. 7 shows a flowchart of another data processing method provided in an embodiment of the present invention, where the method includes the following steps:
step S200, receiving a confirmation request sent by the client, wherein the confirmation request is sent by the client after receiving response messages fed back by a preset number of data nodes in the plurality of data nodes, the response messages fed back by any data node are generated in response to data writing requests sent by the client and received by the client, and the confirmation request comprises confirmation information used for representing a storage block to be confirmed, a storage area to be confirmed and a version number to be confirmed.
In this embodiment, after the client 20 sends the same set of erasure code encoded data to different data nodes 10, the data nodes 10 successfully store the data sent to the client 10 to corresponding target snapshots, and then feed back response messages indicating successful storage to the client 20, according to the difference in the encoding modes of the erasure codes, the client 20 determines, according to the number of received response messages indicating successful storage, whether the data of the same set of erasure code encoded data is successfully stored or not, and for an n + m encoding mode, where n data to be stored and m check data are stored, the client 20 sends n + m data to different data nodes 10 for storage, and if n data nodes 10 feed back response messages indicating successful storage to the client 20, the client 20 confirms that the (n + m) data are successfully stored, because for the data of the same set of erasure code encoded data, as long as any n data of the (n + m) data are successfully stored, the policy of the remaining m data can be restored according to the encoding policy of the erasure codes.
It should be noted that the client 20 only sends an acknowledgement message to the data node 10 feeding back the response message of successful storage. For example, the client 20 sends a write data request to the data nodes 1 to 5, the write addresses are (1 # memory block, 1# memory cell), (2 # memory block, 1# memory cell), (1 # memory block, 2# memory cell), (1 # memory block, 1# memory cell), the preset number is 3 according to the erasure coding rule of 3+2, the data nodes 1, 3, and 5 feed back response messages indicating successful storage to the client 20 and feed back corresponding version numbers 1, 2, and the corresponding storage areas are: 1 storage area, 2 storage area, 1 storage area, the client 20 sends confirmation messages (1 # storage block, 1# storage area, version number 1), (1 # storage block, 2# storage area, version number 2), (1 # storage block, 1# storage area, version number 2) to the data node 1, the data node 3, and the data node 5, respectively.
It should be further noted that, after the client 20 confirms that the data storage is successful, for the data node 10 which is not fed back and successfully stored, the client 20 sends an exception message to the management node 30, where the exception message may carry a storage block and a storage unit in which an exception occurs, and the management node 30 recovers, according to the received exception message, data in the storage block and the storage unit corresponding to the data node 10 which is not fed back and successfully stored, according to the storage block and the storage unit which are successfully stored on the data node 10.
It should be further noted that, the next data writing request may incidentally carry confirmation information representing the to-be-confirmed storage block, the to-be-confirmed storage area, and the to-be-confirmed version number corresponding to the current data writing request, so as to reduce interaction between the client 20 and the data node, and improve the processing efficiency of the distributed storage system.
Step S210, if the state of the to-be-confirmed storage area with the largest version number in the snapshot of the to-be-confirmed storage block is the to-be-confirmed state, updating the state of the to-be-confirmed storage area corresponding to the to-be-confirmed version number to the confirmed state.
In this embodiment, the state of the to-be-confirmed storage area with the largest version number in the snapshot of the to-be-confirmed storage block is to be confirmed, and the state indicates whether the last newly written data has not been confirmed to be successfully written, at this time, the state of the to-be-confirmed storage area corresponding to the to-be-confirmed version number needs to be updated to be a confirmed state, if the state of the to-be-confirmed storage area with the largest version number in the snapshot of the to-be-confirmed storage block is a confirmed state, the state indicates that the last newly written data has been successfully written, at this time, the data in the to-be-confirmed storage area corresponding to the to-be-confirmed version number is the data written before the last newly written, because the last newly written data has already been confirmed to be successfully written, whether the data written before this time has been successfully written, is unimportant, and the previously written data will be finally overwritten by the last newly written data, that is the data that needs to be finally stored in the corresponding storage unit.
The data processing method provided by the embodiment of the invention updates the state of the data in time when the successful storage of the data is determined, and avoids the update of the state of the data written before after the latest newly written data is confirmed to be successfully written, thereby not only ensuring the timely update of the state, but also reducing unnecessary state processing operation.
In this embodiment, if the client 20 does not receive a response message indicating that storage is successful, which is fed back by a preset number of data nodes, within a preset time period, in order to timely release a space occupied by data that is determined to be unsuccessfully written, an embodiment of the present invention further provides a processing manner for this scenario, please continue referring to fig. 7, where the method further includes the following steps:
step S220, if the confirmation request sent by the client is not received within the preset time period, deleting data in the to-be-confirmed storage area corresponding to the to-be-confirmed version number in the snapshot of the to-be-confirmed storage block.
In the present embodiment, the preset period may be a preset timeout duration, for example, the preset period is 10 milliseconds.
It should be noted that, when the number of the data nodes 10 that receive the feedback response message by the client 20 is less than the preset number, it is determined that the data storage at this time fails, at this time, the client 20 reports a message of writing exception to the management node 30, the management node 30 marks the storage block written with exception, for example, as broken, increments the version number of the storage area written successfully by the data node 10 that feeds back the same group of erasure code codes, and waits for data recovery. As a specific embodiment, a separate thread for restoring data may be specifically created in the management node 30, and periodically scan the memory block marked as broken to restore the data in the memory block to the version of the last acknowledgement state.
According to the data processing method provided by the embodiment of the invention, the response message of successful storage fed back by the preset number of data nodes is not received in the preset time period, and the space occupied by the data which is confirmed to be unsuccessfully written is released in time, so that the waste of the storage space in the data nodes is avoided.
In this embodiment, in order to update data in a snapshot to a storage unit of a corresponding storage block in time, an embodiment of the present invention further provides a specific updating manner, please refer to fig. 8, where fig. 8 shows a flowchart of another data processing method provided in the embodiment of the present invention, where the method includes the following steps:
step S300, traversing the snapshot of the first target storage block, and writing the data in the storage area with the maximum version number and the confirmed state in the snapshot of the first target storage block into the corresponding storage unit in the first target storage block.
In this embodiment, the data in the storage area with the maximum version number and the confirmed status in the snapshot of the first target storage block is the data that was written successfully last time.
It should be noted that, since the data node 10 includes a plurality of storage blocks, the method in step S300 is also applicable to processing of other storage blocks, as a specific implementation, the data node 10 may specifically create an independent thread, traverse each storage block in the data node 10, determine whether a snapshot exists in each storage block, and if a snapshot exists, process according to the processing manner in step S300.
The data processing method provided by the embodiment of the invention updates the data in the snapshot to the storage unit of the corresponding storage block in time, ensures that the data to be stored is written into the finally corresponding storage unit in time, and ensures the reliability of the data in the distributed storage system.
In this embodiment, in order to correctly and timely read out data stored in a distributed storage system, an embodiment of the present invention further provides an implementation manner of reading data, please refer to fig. 9, where fig. 9 shows a flowchart of another data processing method provided in the embodiment of the present invention, where the method includes the following steps:
step S400, a data reading request sent by the client is received, wherein the data reading request carries an address to be read, and the address to be read is used for representing a second target storage block for storing data to be read and a second target storage unit in the second target storage block.
In this embodiment, the second target storage block is a storage block used for storing data to be read in a plurality of storage blocks in the data node 10, and the second target storage unit is a storage unit used for storing data to be read in a plurality of storage units in the second target storage block. The address to be read is similar to the address to be written, and details are not described here.
In this embodiment, the client 20 first receives a data reading request sent by a user, and obtains the data distribution data node 10 that the user needs to read, and the second target storage block and the second target storage unit in the data node 10 from the management node 30 according to the data reading request sent by the user.
Step S410, determining a target reading area corresponding to the data to be read according to the address to be read and reading the data to be read from the target reading area.
In this embodiment, since the data to be written is written into the corresponding target snapshot first, and then the data to be written is updated from the target snapshot to the storage unit in the corresponding storage block, the data to be read may be stored in the snapshot of the storage block or in the storage unit in the storage block. The target reading area is used for representing a storage area where data to be read can be read, and the target reading area can be a storage area in a snapshot of a storage block or a storage unit in the storage block.
Step S420, returns the data to be read to the client.
It should be noted that, when data that needs to be read by a user is distributed on a plurality of data nodes 10, the client 20 sends read data requests to the corresponding data nodes 10, splices and merges the data to be read that is fed back by each data node 10 after receiving the data to be read, and finally feeds back the data that needs to be read to the user.
According to the data processing method provided by the embodiment of the invention, the target reading area corresponding to the data to be read can be correctly and timely determined according to the second target storage block and the second target storage unit, so that the data to be read can be timely and correctly read from the target reading area.
On the basis of fig. 9, an embodiment of the present invention further provides a specific implementation manner of determining a target reading area, referring to fig. 10, fig. 10 shows a flowchart of another data processing method provided in the embodiment of the present invention, and step S410 includes the following sub-steps:
in sub-step S4101, if the snapshot does not exist in the second target storage block, the second target storage unit is set as the target read area.
In sub-step S4102, if the snapshot of the second target storage block exists and the state in the second target storage area in the snapshot of the second target storage block is not the confirmed state, the second target storage unit is set as the target read area.
In this embodiment, if a snapshot exists in the second target storage block, the data in the second target storage block in the snapshot is not in a confirmed state, at least the following situations exist: (1) The snapshot is one, and the data in the second target storage area in the snapshot is not updated; (2) The snapshot is one, and the data in the second target storage area in the snapshot is updated but the state of the second target storage area is a to-be-confirmed state; (3) The number of the snapshots is two, and the data in the second target storage area in the two snapshots are not updated; (4) The number of the snapshots is two, and the data in the second target storage area in the two snapshots are updated but the states of the data are to-be-confirmed states.
In sub-step S4103, if two snapshots exist in the second target storage block, and only one of the two snapshots of the second target storage block is in the confirmed state, the second target storage block in the confirmed state is used as the target read area.
In sub-step S4104, if two snapshots exist in the second target storage block, and the states in the second target storage area in the two snapshots of the second target storage block are both confirmed states, the second target storage area with the largest version number is used as the target read area.
The data processing method provided by the embodiment of the invention gives various conditions of the data to be read, and ensures that the data to be read can be correctly and timely read from the target reading area under various conditions.
In order to perform the corresponding steps in the above-described embodiments and various possible implementations, an implementation of the data processing apparatus 100 is given below. Referring to fig. 11, fig. 11 is a block diagram illustrating a data processing apparatus 100 according to an embodiment of the present invention. It should be noted that the basic principle and the resulting technical effect of the data processing apparatus 100 provided in the present embodiment are the same as those of the above embodiments, and for the sake of brief description, no reference is made to this embodiment.
The data processing apparatus 100 includes a receiving module 110, a processing module 120, and a returning module 130.
The receiving module 110 is configured to receive a data writing request sent by a client, where the data writing request carries an address to be written, and the address to be written is used to represent a first target storage block to which data to be written should be written and a first target storage unit in the first target storage block.
As a specific embodiment, the receiving module 110 is further configured to: and receiving a data reading request sent by the client, wherein the data reading request carries a to-be-read address, and the to-be-read address is used for representing a second target storage block for storing to-be-read data and a second target storage unit in the second target storage block.
The processing module 120 is configured to determine, according to the address to be written, a target snapshot to which the data to be written is written, where the target snapshot corresponds to the first target storage block, and the target snapshot includes a storage area, and the storage area corresponds to a storage unit in the first target storage block; and writing the data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to the first target storage unit.
As a specific implementation, the processing module 120 is specifically configured to: if the first target storage block does not have a snapshot or the first target storage block has a snapshot and the data in the first target storage area corresponding to the first target storage unit in the snapshot is updated, newly creating the snapshot of the first target storage block and taking the newly created snapshot as a target snapshot; if the first target storage block has a snapshot and the data in the first target storage area corresponding to the first target storage unit in the snapshot is not updated, taking the snapshot as a target snapshot; and if the first target storage block has two snapshots, determining the target snapshot from the two snapshots according to a preset rule.
As a specific embodiment, each snapshot corresponds to a first target storage block, each snapshot includes a first target storage area corresponding to a first target storage unit, when data in the first target storage area is updated, the first target storage area corresponds to a version number and a state, the state is a to-be-confirmed state or a confirmed state, and the processing module 120 is specifically configured to, when determining a target snapshot from the two snapshots according to a preset rule: if the data stored in the first target storage area in any snapshot is not updated, taking the snapshot corresponding to the first target storage area of which the data is not updated as a target snapshot; if the data stored in the first target storage area in the two snapshots are not updated, taking any one snapshot in the two snapshots as a target snapshot; and if the data stored in the first target storage area in the two snapshots are updated and the state of the first target storage area with the larger version number is the confirmed state, taking the snapshot corresponding to the first target storage area with the smaller version number as the target snapshot.
As a specific embodiment, the processing module 120 is further configured to: and increasing the version number of the first target storage area and setting the state of the first target storage area as the state to be confirmed.
As a specific implementation manner, the number of data nodes is multiple, the client sends multiple data write requests to the multiple data nodes in advance, where data to be written in the multiple data write requests belongs to the same group of erasure correction coded data, and the processing module 120 is further configured to: receiving a confirmation request sent by a client, wherein the confirmation request is sent by the client after the client receives response messages fed back by a preset number of data nodes in a plurality of data nodes, the response message fed back by any data node is generated in response to a data writing request sent by the client and received by the client, and the confirmation request comprises confirmation information used for representing a storage block to be confirmed, a storage area to be confirmed and a version number to be confirmed; and if the state of the to-be-confirmed storage area with the maximum version number in the snapshot of the to-be-confirmed storage block is the to-be-confirmed state, updating the state of the to-be-confirmed storage area corresponding to the to-be-confirmed version number to be the confirmed state.
As a specific embodiment, the processing module 120 is further configured to: and if the confirmation request sent by the client is not received within the preset time period, deleting the data in the storage area to be confirmed corresponding to the version number to be confirmed in the snapshot of the storage block to be confirmed.
As a specific embodiment, the processing module 120 is further configured to: and traversing the snapshot of the first target storage block, and writing the data in the storage area with the maximum version number and the confirmed state in the snapshot of the first target storage block into the corresponding storage unit in the first target storage block.
As a specific implementation, the processing module 120 is further configured to: and determining a target reading area corresponding to the data to be read according to the address to be read and reading the data to be read from the target reading area.
As a specific implementation, the processing module 120 is specifically configured to: if the second target storage block does not have the snapshot, taking the second target storage unit as a target reading area; if the second target storage block has a snapshot and the state in the second target storage area in the snapshot of the second target storage block is not the confirmed state, taking the second target storage unit as a target reading area; if the second target storage block has two snapshots and only one second target storage area in the two snapshots of the second target storage block is in the confirmed state, taking the second target storage area in the confirmed state as a target reading area; and if the second target storage block has two snapshots and the data in the second target storage area in the two snapshots of the second target storage block are both in a confirmed state, taking the second target storage area with the maximum version number as a target reading area.
And a returning module 130, configured to return a response message that the writing is successful to the client.
As a specific embodiment, the return module 130 is further configured to: and returning the data to be read to the client.
The present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data processing method as described above.
In summary, embodiments of the present invention provide a data processing method, an apparatus, a data node, and a storage medium, which are applied to a data node in a distributed storage system, where the data node includes a plurality of storage blocks, each storage block includes a plurality of storage units, the distributed storage system further includes a client, and the data node is in communication connection with the client, and the method includes: receiving a data writing request sent by a client, wherein the data writing request carries an address to be written, and the address to be written is used for representing a first target storage block in which data to be written is to be written and a first target storage unit in the first target storage block; determining a target snapshot to be written in by data to be written according to the address to be written, wherein the target snapshot corresponds to a first target storage block, the target snapshot comprises a storage area, and the storage area corresponds to a storage unit in the first target storage block; writing data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to a first target storage unit; and returning a response message of successful writing to the client. Compared with the prior art, the embodiment of the invention creates the corresponding snapshot for the storage block, temporarily stores the data to be written into the storage block into the snapshot corresponding to the storage block, and avoids the read-write punishment generated when the data to be written is written into the storage block, thereby improving the write performance of the distributed storage system.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A data processing method applied to a data node in a distributed storage system, wherein the data node includes a plurality of storage blocks, each of the storage blocks includes a plurality of storage units, the distributed storage system further includes a client, the data node is communicatively connected to the client, the data node is multiple, and the method includes:
receiving a data writing request sent by the client, wherein the data writing request carries an address to be written, and the address to be written is used for representing a first target storage block in which data to be written is to be written and a first target storage unit in the first target storage block;
determining a target snapshot into which the data to be written is written according to the address to be written, wherein the target snapshot corresponds to the first target storage block, the target snapshot comprises a storage area, and the storage area corresponds to the storage unit in the first target storage block;
writing the data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to the first target storage unit;
returning a response message of successful writing to the client;
the method further comprises the following steps:
the version number of the first target storage area is increased and the state of the first target storage area is set to be a state to be confirmed;
receiving a confirmation request sent by the client, wherein the confirmation request is sent by the client after receiving response messages fed back by a preset number of data nodes in a plurality of data nodes, the response messages fed back by any data node are generated in response to the data writing request sent by the client, the confirmation request comprises confirmation information used for representing a storage block to be confirmed, a storage area to be confirmed and a version number to be confirmed, the client sends a plurality of data writing requests to the plurality of data nodes in advance, and data to be written in the plurality of data writing requests belong to the same group of erasure correction coding data;
and if the state of the storage area to be confirmed with the maximum version number in the snapshot of the storage block to be confirmed is the state to be confirmed, updating the state of the storage area to be confirmed corresponding to the version number to be confirmed into a confirmed state.
2. The data processing method according to claim 1, wherein the step of determining the target snapshot to be written with the data to be written according to the address to be written comprises:
if the first target storage block does not have a snapshot or the first target storage block has a snapshot and the data in the first target storage area corresponding to the first target storage unit in the snapshot is updated, newly creating the snapshot of the first target storage block and taking the newly created snapshot as a target snapshot;
if the first target storage block has a snapshot and the data in the first target storage area corresponding to the first target storage unit in the snapshot is not updated, taking the snapshot as a target snapshot;
and if the first target storage block has two snapshots, determining a target snapshot from the two snapshots according to a preset rule.
3. The data processing method according to claim 2, wherein each of the snapshots corresponds to the first target storage block, each of the snapshots includes a first target storage area corresponding to the first target storage unit, when data in the first target storage area is updated, the first target storage area corresponds to a version number and a status, the status is a status to be confirmed or a status to be confirmed, and the step of determining a target snapshot from the two snapshots according to a preset rule includes:
if the data stored in the first target storage area in any snapshot is not updated, taking the snapshot corresponding to the first target storage area with the data not updated as a target snapshot;
if the data stored in the first target storage area in the two snapshots are not updated, taking any one snapshot in the two snapshots as a target snapshot;
and if the data stored in the first target storage area in the two snapshots are updated, and when the state of the first target storage area with a large version number is a confirmation state, taking the snapshot corresponding to the first target storage area with a small version number as a target snapshot.
4. The data processing method of claim 1, wherein the method further comprises:
and if the confirmation request sent by the client is not received within a preset time period, deleting the data in the storage area to be confirmed, corresponding to the version number to be confirmed, in the snapshot of the storage block to be confirmed.
5. The data processing method of claim 1, wherein the method further comprises:
and traversing the snapshot of the first target storage block, and writing the data in the storage area with the maximum version number and the confirmed state in the snapshot of the first target storage block into the corresponding storage unit in the first target storage block.
6. The data processing method of claim 1, wherein the method further comprises:
receiving a data reading request sent by the client, wherein the data reading request carries a to-be-read address, and the to-be-read address is used for representing a second target storage block for storing to-be-read data and a second target storage unit in the second target storage block;
determining a target reading area corresponding to the data to be read according to the address to be read and reading the data to be read from the target reading area;
and returning the data to be read to the client.
7. The data processing method according to claim 6, wherein there are at most two snapshots in the second target storage block, each snapshot in the second target storage block includes a second target storage area corresponding to the second target storage unit, when data in the second target storage area is updated, the second target storage area corresponds to a version number and a status, and the step of determining a target read area corresponding to the data to be read according to the address to be read includes:
if the second target storage block does not have the snapshot, taking the second target storage unit as the target reading area;
if the second target storage block has a snapshot and the state in the second target storage area in the snapshot of the second target storage block is not a confirmation state, taking the second target storage unit as the target reading area;
if the second target storage block has two snapshots and only one second target storage area in the two snapshots of the second target storage block is in a confirmed state, taking the second target storage area in the confirmed state as the target reading area;
and if the second target storage block has two snapshots and the data in the second target storage area in the two snapshots of the second target storage block are both in a confirmed state, taking the second target storage area with the largest version number as the target reading area.
8. A data processing apparatus, applied to a data node in a distributed storage system, where the data node includes a plurality of storage blocks, each of the storage blocks includes a plurality of storage units, the distributed storage system further includes a client, the data node is communicatively connected to the client, and the data node is multiple, the apparatus includes:
a receiving module, configured to receive a data writing request sent by the client, where the data writing request carries an address to be written, and the address to be written is used to represent a first target storage block to which data to be written should be written and a first target storage unit in the first target storage block;
a processing module to: determining a target snapshot into which the data to be written is written according to the address to be written, wherein the target snapshot corresponds to the first target storage block, the target snapshot comprises a storage area, and the storage area corresponds to the storage unit in the first target storage block; writing the data to be written into a first target storage area in the storage areas, wherein the first target storage area corresponds to the first target storage unit;
the processing module is further configured to: the version number of the first target storage area is increased and the state of the first target storage area is set to be a state to be confirmed;
the processing module is further configured to: receiving a confirmation request sent by the client, wherein the confirmation request is sent by the client after receiving response messages fed back by a preset number of data nodes in a plurality of data nodes, the response message fed back by any data node is generated in response to the data writing request sent by the client, the confirmation request comprises confirmation information used for representing a storage block to be confirmed, a storage area to be confirmed and a version number to be confirmed, the client sends a plurality of data writing requests to the plurality of data nodes in advance, and data to be written in the plurality of data writing requests belong to the same group of erasure correction coding data;
if the state of the storage area to be confirmed with the maximum version number in the snapshot of the storage block to be confirmed is the state to be confirmed, updating the state of the storage area to be confirmed corresponding to the version number to be confirmed into a confirmed state;
and the return module is used for returning a response message of successful writing to the client.
9. A data node, characterized in that the data node comprises:
one or more processors;
memory for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010758439.6A CN111857602B (en) | 2020-07-31 | 2020-07-31 | Data processing method, data processing device, data node and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010758439.6A CN111857602B (en) | 2020-07-31 | 2020-07-31 | Data processing method, data processing device, data node and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111857602A CN111857602A (en) | 2020-10-30 |
| CN111857602B true CN111857602B (en) | 2022-10-28 |
Family
ID=72953518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010758439.6A Active CN111857602B (en) | 2020-07-31 | 2020-07-31 | Data processing method, data processing device, data node and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111857602B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988683A (en) * | 2021-02-07 | 2021-06-18 | 北京金山云网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN113014662A (en) * | 2021-03-11 | 2021-06-22 | 联想(北京)有限公司 | Data processing method and storage system based on NVMe-oF protocol |
| CN113253925B (en) * | 2021-04-30 | 2022-08-30 | 新华三大数据技术有限公司 | Method and device for optimizing read-write performance |
| CN113515531B (en) * | 2021-05-08 | 2022-12-02 | 重庆紫光华山智安科技有限公司 | Data access method, device, client and storage medium |
| CN114115732A (en) * | 2021-11-10 | 2022-03-01 | 深圳Tcl新技术有限公司 | Data processing method, device and system |
Family Cites Families (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7363444B2 (en) * | 2005-01-10 | 2008-04-22 | Hewlett-Packard Development Company, L.P. | Method for taking snapshots of data |
| CN100447758C (en) * | 2005-12-14 | 2008-12-31 | 英业达股份有限公司 | Snapshot management method for logic disk |
| US8135930B1 (en) * | 2008-07-14 | 2012-03-13 | Vizioncore, Inc. | Replication systems and methods for a virtual computing environment |
| US8306950B2 (en) * | 2010-08-26 | 2012-11-06 | International Business Machines Corporation | Managing data access requests after persistent snapshots |
| CN102779080B (en) * | 2012-06-18 | 2015-05-27 | 上海爱数软件有限公司 | Method for generating snapshot, method and device for data recovery by using snapshot |
| JP2015532985A (en) * | 2012-09-06 | 2015-11-16 | ピーアイ−コーラル、インク． | Large-scale data storage and delivery system |
| CN105095016B (en) * | 2014-05-16 | 2018-05-18 | 北京云巢动脉科技有限公司 | A kind of disk snapshot rollback method and device |
| JP6607941B2 (en) * | 2014-12-27 | 2019-11-20 | 華為技術有限公司 | Data processing method, apparatus, and system |
| CN106294205B (en) * | 2015-05-15 | 2019-12-10 | 北京国双科技有限公司 | Cache data processing method and device |
| US20170228285A1 (en) * | 2016-02-10 | 2017-08-10 | SwiftStack, Inc. | Data durability in stored objects |
| US20180165380A1 (en) * | 2016-03-29 | 2018-06-14 | Hitachi, Ltd. | Data processing system and data processing method |
| US10452290B2 (en) * | 2016-12-19 | 2019-10-22 | Pure Storage, Inc. | Block consolidation in a direct-mapped flash storage system |
| US10261719B2 (en) * | 2017-01-31 | 2019-04-16 | Hewlett Packard Enterprise Development Lp | Volume and snapshot replication |
| US10360214B2 (en) * | 2017-10-19 | 2019-07-23 | Pure Storage, Inc. | Ensuring reproducibility in an artificial intelligence infrastructure |
| CN109144416B (en) * | 2018-08-03 | 2020-04-28 | 华为技术有限公司 | Method and device for querying data |
| CN110209351B (en) * | 2019-05-10 | 2021-02-19 | 星辰天合（北京）数据科技有限公司 | Distributed storage data processing method and device |
-
2020
- 2020-07-31 CN CN202010758439.6A patent/CN111857602B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111857602A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111857602B (en) | Data processing method, data processing device, data node and storage medium | |
| CN110018783B (en) | Data storage method, device and system | |
| US7882286B1 (en) | Synchronizing volumes for replication | |
| US11698728B2 (en) | Data updating technology | |
| US11307776B2 (en) | Method for accessing distributed storage system, related apparatus, and related system | |
| US11397668B2 (en) | Data read/write method and apparatus, and storage server | |
| CN113515531B (en) | Data access method, device, client and storage medium | |
| CN103885895A (en) | Write Performance in Fault-Tolerant Clustered Storage Systems | |
| CN106708653B (en) | Mixed tax big data security protection method based on erasure code and multiple copies | |
| WO2017088572A1 (en) | Data processing method, device, and system | |
| WO2023206968A1 (en) | Data storage method and system, and computer readable storage medium | |
| US20080133967A1 (en) | Distributed object sharing system and method thereof | |
| CN114647383A (en) | Data access method, device, storage node and storage medium | |
| CN112214175A (en) | Data processing method, data processing device, data node and storage medium | |
| CN111857603B (en) | Data processing method and related device | |
| CN107329699B (en) | Erasure rewriting method and system | |
| CN116560795A (en) | Replication method and related device of distributed block storage system | |
| CN115878381A (en) | Data recovery method and device based on SRM disc, storage medium and electronic device | |
| CN116028253A (en) | Solid state disk error correction optimization method, system, equipment and medium | |
| CN107045426B (en) | Multi-copy reading method and system | |
| CN115309336A (en) | Data writing method, cache information updating method and related device | |
| CN115470041A (en) | Data disaster recovery management method and device | |
| CN112527561B (en) | Data backup method and device based on Internet of things cloud storage | |
| CN115248745A (en) | Data processing method and device | |
| CN108174136B (en) | Cloud disk video coding storage method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |