CN107102918B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN107102918B CN107102918B CN201710386160.8A CN201710386160A CN107102918B CN 107102918 B CN107102918 B CN 107102918B CN 201710386160 A CN201710386160 A CN 201710386160A CN 107102918 B CN107102918 B CN 107102918B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- storage unit
- destination
- data stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000000034 method Methods 0.000 claims abstract description 16
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
- G06F11/2064—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring while ensuring consistency
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
- G06F11/2069—Management of state, configuration or failover
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a data processing method and a device, wherein the method comprises the following steps: after the data flow of the first data reaches a forwarding layer, judging whether the node is a destination node of the data flow of the first data; the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data; if not, determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and sending the data stream of the first data to a cache layer by the destination node; if so, the data stream of the first data is sent to the cache layer, and the consistency of the data is ensured by the invention.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus.
Background
Most of the existing storage systems are dual-node storage systems, and when one node fails, another node can be used for caching data, so that normal operation of services is guaranteed.
In recent years, in order to further improve the security of data caching, a multi-node storage system has been proposed in addition to a dual-node storage system, and the multi-node storage system continues to cache data in a dual-node manner. Specifically, when a certain data is cached, the data stream of the data is forwarded to any node, so that the technical problem of data inconsistency occurs.
Disclosure of Invention
In view of this, the present invention provides a data processing method and apparatus, so as to solve the technical problem of inconsistent data in the prior art.
In order to achieve the purpose, the invention provides the following technical scheme:
a method of data processing, comprising:
after the data flow of the first data reaches a forwarding layer, judging whether the node is a destination node of the data flow of the first data; the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data;
if not, determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and sending the data stream of the first data to a cache layer by the destination node;
and if so, sending the data stream of the first data to a cache layer.
Preferably, before the determining whether the node is a destination node of the data stream of the first data, the method further includes:
judging whether the node is a node in a node group to which the first storage unit belongs;
if yes, triggering the step of judging whether the node is a destination node of the data flow of the first data;
if not, triggering the step of determining the destination node of the first data flow.
Preferably, after sending the data stream of the first data to the cache layer, the method further includes:
when the node is a node to which the first storage unit belongs, the first data is mirrored to a cooperative node of the node to which the first data belongs;
when the node is a cooperative node of the first storage unit, mirroring the first data to a front node of the node;
and the front node of the node is the node to which the first storage unit belongs.
Preferably, the determining the destination node of the first data flow includes:
and selecting a node with large data processing capacity as a destination node from the affiliated node and the cooperative node of the affiliated node.
Preferably, the selecting a node with a large data processing capability from the cooperative node of the node and the node as a destination node includes:
determining a node which carries out data processing last time from the affiliated node and a cooperative node of the affiliated node;
and determining another node except the node which carries out data processing last time as a destination node.
A data processing apparatus comprising:
a first determining unit, configured to determine whether a node is a destination node of a data stream of first data after the data stream of the first data reaches a forwarding layer; the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data;
a determining and forwarding unit, configured to determine, when a node is not a destination node of a data stream of the first data, the destination node of the data stream of the first data, forward the data stream of the first data to the destination node, and send, by the destination node, the data stream of the first data to a cache layer;
and a first sending unit, configured to send the data stream of the first data to a cache layer when the node is a destination node of the data stream of the first data.
Preferably, the method further comprises the following steps:
the second judging unit is used for judging whether the node is a node in the node group to which the first storage unit belongs or not after the data stream of the first data reaches the forwarding layer;
a first triggering unit, configured to, when a local node is a node in a node group to which a first storage unit belongs, trigger the first determining unit to perform an operation of determining whether the local node is a destination node of a data stream of the first data;
and the second triggering unit is used for triggering the determining and forwarding unit to execute the operation of determining the destination node of the first data stream when the node is not the node in the node group to which the first storage unit belongs.
Preferably, the method further comprises the following steps:
a first mirroring unit, configured to mirror the first data to a cooperative node of the node to which the node belongs when the node is the node to which the first storage unit belongs;
a second mirroring unit, configured to mirror the first data to a front node of a local node when the local node is a cooperative node of a node to which the first storage unit belongs;
and the front node of the node is the node to which the first storage unit belongs.
Preferably, the determining the forwarding unit includes:
a first determining module, configured to select, as a destination node, a node with a large data processing capability from cooperative nodes of the node and the node when the node is not the destination node of the data stream of the first data;
and the first forwarding module is used for forwarding the data stream of the first data to the destination node, and the destination node sends the data stream of the first data to a cache layer.
Preferably, the first determining module is specifically configured to determine a node which has performed data processing last time from the cooperative nodes of the node to which the node belongs and the node to which the node belongs, so as to determine another node except the node which has performed data processing last time as a destination node.
As can be seen from the foregoing technical solutions, compared with the prior art, the present invention discloses a data processing method, specifically, after a data stream of first data reaches a forwarding layer, determining whether a node is a destination node of the data stream of the first data, where the destination node includes a node to which a first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data; if not, determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and finally sending the data stream of the first data to the cache layer by the destination node. Therefore, the data stream of the first data in the application can be forwarded to the destination node, and is cached to the cache layer by the destination node, so that the consistency of the data is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic structural diagram of a multi-node memory system according to the present disclosure;
FIG. 2 is a flowchart illustrating a data processing method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a data processing method according to a second embodiment of the present invention;
fig. 4 is a schematic flow chart of a data processing method according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data processing apparatus according to a fifth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data processing apparatus according to a sixth embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
The present invention discloses a data processing method, which is applied to a multi-node storage system, and specifically, fig. 1 shows a schematic structural diagram of the multi-node storage system disclosed by the present invention, as shown in fig. 1, a client Host is connected with N nodes, and the N nodes form a node group, wherein one node is a cooperative node of a previous node adjacent to the node and a previous node of a next node adjacent to the node. That is, two adjacent nodes are in a single-party cooperative relationship and not in a mutual cooperative relationship. That is, Node1 is a cooperative Node of Node0, which is a front Node of Node 2; node2 is a cooperative Node of Node1, a front Node of Node3, … and so on.
It should be noted that the multi-node storage system includes at least one node group, and fig. 1 illustrates that the client corresponds to one node group, and of course, the client may correspond to a plurality of node groups, which is not limited in the present invention. And each node group includes at least two nodes.
Fig. 2 shows a flow diagram of a data processing method, as shown in fig. 2, comprising the steps of:
step 201: after the data flow of the first data reaches a forwarding layer, judging whether the node is a destination node of the data flow of the first data; if not, go to step 202; if yes, go to step 203;
the node is designated as a node on its forwarding layer that receives a data stream of the first data.
The destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs.
The first storage unit is a write-in unit of the first data, specifically, the first storage unit is a LUN (logical unit Number), and one LUN may be composed of one disk, several disks, or a partial space on one disk.
The node to which the first storage unit belongs is a node owning the first storage unit, and the node is a designated node of the first storage unit and is used for writing data needing to be written into the first storage unit. As can be seen from the foregoing description of the multi-node storage system, the node to which the first storage unit belongs and the cooperative node of the node to which the first storage unit belongs are in a unidirectional cooperative relationship, and the node to which the first storage unit belongs is a preceding node of the cooperative node of the node to which the first storage unit belongs.
When a data stream of first data sent by a client reaches a forwarding layer of a certain node, the node judges whether the node is a destination node of the data stream of the first data.
Specifically, each node in the multi-node storage system stores the node index of the node and the bitmap information of all the nodes, so that whether the node is the destination node of the data stream of the first data can be determined based on the node index of the node and the bitmap information of all the nodes. For example, the bitmap information is a 32-bit integer value, i.e., for nodes 0-31. If the node index of the node is the nth bit, it indicates that the data stream of the first storage unit received on the node N needs to be forwarded if the nth position 1 of the bitmap information is set to 0, that is, the node N is not a destination node of the data stream of the first data, and indicates that the data stream of the first storage unit received on the node N does not need to be forwarded if the nth position 1 of the bitmap information is set to 0, that is, the node N is a destination node of the data stream of the first data. That is, the bitmap information has bits of 1 except for the bit of the node to which the data stream of the first data belongs of the first storage unit and the position 0 of the cooperative node.
Step 202: determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and sending the data stream of the first data to a cache layer by the destination node;
when it is determined that the node is not the destination node of the data stream of the first data, one node may be selected from the cooperative nodes of the node to which the first storage unit belongs and the node to which the first storage unit belongs as the destination node, so that the data stream of the first data is forwarded to the forwarding layer of the destination node, and the destination node sends the data stream of the first data to the cache layer of the destination node.
Step 203: and sending the data stream of the first data to a buffer layer.
When the node is determined to be the destination node of the data stream of the first data, the data stream of the first data is directly sent to the cache layer of the node by the node. It should be noted that there may be other data processing layers between the forwarding layer and the cache layer of the first node, however, the data stream of the first data is finally sent to the cache layer, that is, sending the data stream of the first data to the cache layer does not necessarily refer to sending the data stream of the first data from the forwarding layer to the cache layer directly, but may refer to sending the data stream of the first data from the forwarding layer to other data processing layers, and then sending the data stream of the first data to the cache layer finally after data processing.
In this embodiment, after the data stream of the first data reaches the forwarding layer, it is determined whether the node is a destination node of the data stream of the first data, where the destination node includes a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data; if not, determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and finally sending the data stream of the first data to the cache layer by the destination node. Therefore, the data stream of the first data in the application can be forwarded to the destination node, and is cached to the cache layer by the destination node, so that the consistency of the data is ensured.
Example two
The second embodiment of the present invention further discloses a data processing method, as shown in fig. 3, the method includes the following steps:
step 301: after the data flow of the first data reaches a forwarding layer, judging whether the node is a node in a node group to which the first storage unit belongs; if yes, go to step 302; if not, go to step 303;
the first storage unit is a write unit of first data and has a node to which the first storage unit belongs. In the multi-node storage system, each node belongs to a node group, and correspondingly, the node group to which the first storage unit belongs is a node group of the nodes having the first storage unit.
Specifically, each node in the multi-node storage system stores the node index of the node and the bitmap information of all the nodes, so that whether the node is a node in the node group to which the first storage unit belongs can be determined based on the node index of the node and the bitmap information of all the nodes.
Step 302: judging whether the node is a destination node of the data flow of the first data; if not, go to step 303, if yes, go to step 304;
step 303: determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and sending the data stream of the first data to a cache layer by the destination node;
as a specific implementation manner, determining the destination node of the first data flow includes: and selecting a node with high data processing capacity as a destination node from the affiliated node and cooperative nodes of the affiliated node.
Specifically, selecting a node with high data processing capability from the node and a cooperative node of the node as a destination node includes: and determining a node which carries out data processing last time from the affiliated node and cooperative nodes of the affiliated node, thereby determining another node except the node which carries out data processing last time as a destination node.
Determining a node which carries out data processing last time from the node which belongs to the first storage unit and the cooperative node of the node which belongs to the first storage unit, and if the node which carries out data processing last time is the node which belongs to the first storage unit, determining the cooperative node of the node which belongs to the first storage unit as a target node; and if the node which carries out data processing last time is a cooperative node of the node which belongs to the first storage unit, determining that the node which belongs to the first storage unit is a destination node.
Step 304: and sending the data stream of the first data to a cache layer.
In this embodiment, after the data stream of the first data reaches the forwarding layer, it is determined whether the node is a node in the node group to which the first storage unit belongs, if the node is a node in the node group to which the first storage unit belongs, it is determined whether the node is a destination node of the data stream of the first data, and if the node is not a node in the node group to which the first storage unit belongs, the destination node of the data stream of the first data may be directly determined, so that the data stream of the first data is sent to the cache layer by the destination node, and consistency of the data is ensured.
EXAMPLE III
The third embodiment of the present invention further discloses a data processing method, as shown in fig. 4, the method includes the following steps:
step 401: after the data flow of the first data reaches a forwarding layer, judging whether the node is a destination node of the data flow of the first data; if not, go to step 402; if yes, go to step 403;
the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs.
The first storage unit is a writing unit of first data.
Step 402: determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and sending the data stream of the first data to a cache layer by the destination node;
when it is determined that the node is not the destination node of the data stream of the first data, one node may be selected from the cooperative nodes of the node to which the first storage unit belongs and the node to which the first storage unit belongs as the destination node, so that the data stream of the first data is forwarded to the forwarding layer of the destination node, and the destination node sends the data stream of the first data to the cache layer of the destination node.
Step 403: sending a data stream of first data to a cache layer;
step 404: when the node is a node of the first storage unit, mirroring the first data to a cooperative node of the node;
step 405: and when the node is a cooperative node of the first storage unit, mirroring the first data to a front node of the node.
The front node of the node is a node to which the first storage unit belongs. When the node is the cooperative node of the first storage unit, the mirroring of the first data to the front node of the node is equivalent to the mirroring of the first data to the node of the first storage unit.
In this embodiment, after a data stream reaches a cache layer, when a node is a node to which the first storage unit belongs, the first data is mirrored to a cooperative node of the node to which the data stream belongs, and when the node is a cooperative node of the node to which the first storage unit belongs, the first data is mirrored to a front node of the node, so that when a node fails, it is ensured that data can be cached by another node. In addition, since the first data is stored in the node to which the first storage unit belongs and the cooperative node of the node to which the first storage unit belongs, it is possible to ensure that the data sent to the first storage unit via the node to which the first storage unit belongs is the latest data.
Corresponding to the data processing method, the invention also provides a data processing device which is applied to the multi-node storage system. Specifically, the following examples are given for a detailed description:
example four
An embodiment of the present invention fourth discloses a data processing apparatus, as shown in fig. 5, the apparatus includes: a first judging unit 501, a determining and forwarding unit 502 and a first sending unit 503; wherein:
a first determining unit 501, configured to determine whether a node is a destination node of a data stream of first data after the data stream of the first data reaches a forwarding layer;
the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data.
The node to which the first storage unit belongs is a node owning the first storage unit, and the node is a designated node of the first storage unit and is used for writing data needing to be written into the first storage unit. As can be seen from the foregoing description of the multi-node storage system, the node to which the first storage unit belongs and the cooperative node of the node to which the first storage unit belongs are in a unidirectional cooperative relationship, and the node to which the first storage unit belongs is a preceding node of the cooperative node of the node to which the first storage unit belongs.
The first determining unit 501 is configured to determine whether a node is a destination node of a data stream of first data after the data stream of the first data sent by the client reaches a forwarding layer of the node. Specifically, each node in the multi-node storage system stores information related to the first storage unit, the information including a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs, so that the first judgment unit can determine whether the node is a destination node based on the information.
A determining and forwarding unit 502, configured to determine, when a node is not a destination node of a data stream of the first data, a destination node of the data stream of the first data, forward the data stream of the first data to the destination node, and send, by the destination node, the data stream of the first data to a cache layer;
when it is determined that the node is not the destination node of the data stream of the first data, the determining and forwarding unit may select one node from the cooperative nodes of the node to which the first storage unit belongs and the node to which the first storage unit belongs as the destination node, so as to forward the data stream of the first data to a forwarding layer of the destination node, and the destination node sends the data stream of the first data to a cache layer of the destination node.
Specifically, as an implementation manner, determining the forwarding unit 502 includes: the device comprises a first determining module and a first forwarding module. The first determining module is configured to select a node with a large data processing capability as a destination node from cooperative nodes of the node and the node when the node is not the destination node of the data stream of the first data; and the first forwarding module is used for forwarding the data stream of the first data to the destination node, and the destination node sends the data stream of the first data to a cache layer.
The first determining module is specifically configured to determine a node that has performed data processing last time from the cooperative node of the node and the node, and thereby determine another node except the node that has performed data processing last time as a destination node. That is, if the node which has performed data processing last time is the node to which the first storage unit belongs, the first determining module determines that the cooperative node of the node to which the first storage unit belongs is the destination node; if the node which carries out data processing last time is a cooperative node of the node which the first storage unit belongs to, the first determining module determines that the node which the first storage unit belongs to is a destination node.
A first sending unit 503, configured to send the data stream of the first data to the cache layer when the node is a destination node of the data stream of the first data.
In this embodiment, after the data stream of the first data reaches the forwarding layer, it is determined whether the node is a destination node of the data stream of the first data, where the destination node includes a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data; if not, determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and finally sending the data stream of the first data to the cache layer by the destination node. Therefore, the data stream of the first data in the application can be forwarded to the destination node, and is cached to the cache layer by the destination node, so that the consistency of the data is ensured.
EXAMPLE five
An embodiment of the present invention fifth discloses a data processing apparatus, as shown in fig. 6, the apparatus includes: a second judging unit 601, a first triggering unit 602, a second triggering unit 603, a first judging unit 604, a determining and forwarding unit 605, and a first sending unit 606; wherein:
a second determining unit 601, configured to determine whether the node is a node in the node group to which the first storage unit belongs after the data stream of the first data reaches the forwarding layer;
the first storage unit is a write unit of first data and has a node to which the first storage unit belongs. In the multi-node storage system, each node belongs to a node group, and correspondingly, the node group to which the first storage unit belongs is a node group of the nodes having the first storage unit.
Specifically, each node in the multi-node storage system stores information related to the first storage unit, where the information includes a node to which the first storage unit belongs, a cooperative node of the node to which the first storage unit belongs, and a node group identifier to which the first storage unit belongs; each node in the multi-node storage system also stores the identifier of the node group to which the node belongs. Therefore, the second determination unit may determine whether or not the node is a node in the node group to which the first storage unit belongs by determining whether or not the identifier of the node group to which the node belongs coincides with the identifier of the node group to which the first storage unit belongs.
A first triggering unit 602, configured to, when a local node is a node in a node group to which a first storage unit belongs, trigger a first determining unit 604 to perform an operation of determining whether the local node is a destination node of a data stream of the first data;
a second triggering unit 603, configured to trigger the determining and forwarding unit 605 to perform an operation of determining a destination node of the first data stream when the node is not a node in the node group to which the first storage unit belongs;
a first judging unit 604, configured to judge whether the node is a destination node of a data stream of the first data;
the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of first data.
A determining and forwarding unit 605, configured to determine a destination node of the data stream of the first data when the node is not the destination node of the data stream of the first data, forward the data stream of the first data to the destination node, and send the data stream of the first data to the cache layer by the destination node;
a first sending unit 606, configured to send the data stream of the first data to the cache layer when the node is a destination node of the data stream of the first data.
In this embodiment, after a data stream of first data reaches a forwarding layer, it is first determined whether a node is a node in a node group to which a first storage unit belongs, and if the node is a node in the node group to which the first storage unit belongs, it is then determined whether the node is a destination node of the data stream of the first data.
EXAMPLE six
An embodiment of the present invention sixth discloses a data processing apparatus, as shown in fig. 7, the apparatus includes: a first judging unit 701, a determining and forwarding unit 702, a first sending unit 703, a first mirroring unit 704 and a second mirroring unit 705; wherein:
a first determining unit 701, configured to determine whether a node is a destination node of a data stream of first data after the data stream of the first data reaches a forwarding layer;
the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of first data.
A determining and forwarding unit 702, configured to determine, when the node is not a destination node of a data stream of the first data, the destination node of the data stream of the first data, forward the data stream of the first data to the destination node, and send, by the destination node, the data stream of the first data to the cache layer;
a first sending unit 703, configured to send, when the node is a destination node of a data stream of first data, the data stream of the first data to the cache layer;
a first mirroring unit 704, configured to mirror the first data to a cooperative node of the node when the node is the node to which the first storage unit belongs;
a second mirroring unit 705, configured to mirror the first data to a front node of the local node when the local node is a cooperative node of the node to which the first storage unit belongs;
and the front node of the node is the node to which the first storage unit belongs. When the node is the cooperative node of the first storage unit, the mirroring of the first data to the front node of the node is equivalent to the mirroring of the first data to the node of the first storage unit.
In this embodiment, after a data stream reaches a cache layer, when a node is a node to which the first storage unit belongs, the first data is mirrored to a cooperative node of the node to which the data stream belongs, and when the node is a cooperative node of the node to which the first storage unit belongs, the first data is mirrored to a front node of the node, so that when a node fails, it is ensured that data can be cached by another node. In addition, since the first data is stored in the node to which the first storage unit belongs and the cooperative node of the node to which the first data belongs, it can be ensured that the data to which the first data is issued to the first storage unit by the node to which the first storage unit belongs is the latest data.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data processing method is applied to a multi-node storage system, wherein each node in the multi-node storage system stores a node index of the node and bitmap information of all nodes, and the method comprises the following steps:
after the data flow of the first data reaches a forwarding layer, judging whether the node is a destination node of the data flow of the first data; wherein, judging whether the node is the destination node of the data flow of the first data includes: judging whether the node is a destination node of the data stream of the first data or not based on the node index of the node and the bitmap information of all the nodes; the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data;
if not, determining a destination node of the data stream of the first data, forwarding the data stream of the first data to the destination node, and sending the data stream of the first data to a cache layer by the destination node;
and if so, sending the data stream of the first data to a cache layer.
2. The method of claim 1, wherein before determining whether the node is a destination node of the data flow of the first data, the method further comprises:
judging whether the node is a node in a node group to which the first storage unit belongs;
if yes, triggering the step of judging whether the node is a destination node of the data flow of the first data;
if not, triggering the step of determining the destination node of the data flow of the first data.
3. The method of claim 1, wherein after sending the data stream of the first data to a buffer layer, further comprising:
when the node is a node to which the first storage unit belongs, the first data is mirrored to a cooperative node of the node to which the first data belongs;
when the node is a cooperative node of the first storage unit, mirroring the first data to a front node of the node;
and the front node of the node is the node to which the first storage unit belongs.
4. The method of claim 1, wherein determining the destination node of the data flow of the first data comprises:
and selecting a node with large data processing capacity as a destination node from the affiliated node and the cooperative node of the affiliated node.
5. The method according to claim 4, wherein said selecting a node with a large data processing capability from said cooperative node of said home node and said home node as a destination node comprises:
determining a node which carries out data processing last time from the affiliated node and a cooperative node of the affiliated node;
and determining another node except the node which carries out data processing last time as a destination node.
6. A data processing apparatus, applied to a multi-node storage system in which each node stores a node index of the node and bitmap information of all nodes, comprising:
a first determining unit, configured to determine whether a node is a destination node of a data stream of first data after the data stream of the first data reaches a forwarding layer; wherein, judging whether the node is the destination node of the data flow of the first data includes: judging whether the node is a destination node of the data stream of the first data or not based on the node index of the node and the bitmap information of all the nodes; the destination node comprises a node to which the first storage unit belongs and a cooperative node of the node to which the first storage unit belongs; the first storage unit is a writing unit of the first data;
a determining and forwarding unit, configured to determine, when a node is not a destination node of a data stream of the first data, the destination node of the data stream of the first data, forward the data stream of the first data to the destination node, and send, by the destination node, the data stream of the first data to a cache layer;
and a first sending unit, configured to send the data stream of the first data to a cache layer when the node is a destination node of the data stream of the first data.
7. The apparatus of claim 6, further comprising:
the second judging unit is used for judging whether the node is a node in the node group to which the first storage unit belongs or not after the data stream of the first data reaches the forwarding layer;
a first triggering unit, configured to, when a local node is a node in a node group to which a first storage unit belongs, trigger the first determining unit to perform an operation of determining whether the local node is a destination node of a data stream of the first data;
and the second triggering unit is used for triggering the determining and forwarding unit to execute the operation of determining the destination node of the data flow of the first data when the node is not the node in the node group to which the first storage unit belongs.
8. The apparatus of claim 6, further comprising:
a first mirroring unit, configured to mirror the first data to a cooperative node of the node to which the node belongs when the node is the node to which the first storage unit belongs;
a second mirroring unit, configured to mirror the first data to a front node of a local node when the local node is a cooperative node of a node to which the first storage unit belongs;
and the front node of the node is the node to which the first storage unit belongs.
9. The apparatus of claim 6, wherein the determining the forwarding unit comprises:
a first determining module, configured to select, as a destination node, a node with a large data processing capability from cooperative nodes of the node and the node when the node is not the destination node of the data stream of the first data;
and the first forwarding module is used for forwarding the data stream of the first data to the destination node, and the destination node sends the data stream of the first data to a cache layer.
10. The apparatus according to claim 9, wherein the first determining module is specifically configured to determine a node that has performed data processing last time from the cooperative nodes of the node and the node, and thereby determine another node except the node that has performed data processing last time as a destination node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710386160.8A CN107102918B (en) | 2017-05-26 | 2017-05-26 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710386160.8A CN107102918B (en) | 2017-05-26 | 2017-05-26 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107102918A CN107102918A (en) | 2017-08-29 |
| CN107102918B true CN107102918B (en) | 2020-06-16 |
Family
ID=59670072
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710386160.8A Active CN107102918B (en) | 2017-05-26 | 2017-05-26 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107102918B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968453A (en) * | 2018-09-28 | 2020-04-07 | 杭州海康威视系统技术有限公司 | Data storage method and device |
| CN109460187B (en) * | 2018-11-09 | 2022-05-03 | 郑州云海信息技术有限公司 | Verification method and verification terminal for data consistency of qcow2 file |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101562543B (en) * | 2009-05-25 | 2013-07-31 | 阿里巴巴集团控股有限公司 | Cache data processing method and processing system and device thereof |
| GB2486780B (en) * | 2009-10-19 | 2015-06-03 | Ibm | Dynamic resource allocation for distributed cluster storage network |
| CN102594698A (en) * | 2012-03-12 | 2012-07-18 | 中国人民解放军总参谋部第六十三研究所 | DTN asynchronous routing algorithm based on node position projection |
| CN104008111B (en) * | 2013-02-27 | 2019-02-15 | 深圳市腾讯计算机系统有限公司 | A kind of memory management method and device of data |
| US9069681B1 (en) * | 2013-03-20 | 2015-06-30 | Google Inc. | Real-time log joining on a continuous stream of events that are approximately ordered |
| EP2978180A1 (en) * | 2014-07-22 | 2016-01-27 | Alcatel Lucent | Signaling for transmission of coherent data flow within packet-switched network |
| CN105721398B (en) * | 2014-12-04 | 2019-07-26 | 深圳市金蝶天燕中间件股份有限公司 | The method and device of data access |
| CN105577547B (en) * | 2016-01-18 | 2019-03-22 | 重庆邮电大学 | Multi-Qos route selection method is based in a kind of mobile ad-hoc network |
| CN106156328B (en) * | 2016-07-06 | 2019-05-07 | 中国银行股份有限公司 | A kind of bank's running log data monitoring method and system |
-
2017
- 2017-05-26 CN CN201710386160.8A patent/CN107102918B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107102918A (en) | 2017-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108810041B (en) | Data writing and capacity expansion method and device for distributed cache system | |
| CN102681794B (en) | Method and system for realizing redundant array protection of a disk based on double controllers | |
| US10469577B2 (en) | Caching method and system based on cache cluster | |
| CN103268318B (en) | A kind of distributed key value database system of strong consistency and reading/writing method thereof | |
| US8209587B1 (en) | System and method for eliminating zeroing of disk drives in RAID arrays | |
| CA2868247C (en) | Data sending method, data receiving method, and storage device | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| US10353870B2 (en) | Tracking structure for data replication synchronization | |
| US20210075665A1 (en) | Implementing switchover operations between computing nodes | |
| CN107329704B (en) | Cache mirroring method and controller | |
| EP3358466A1 (en) | Data replication method and storage system | |
| CN110651246B (en) | Data reading and writing method and device and storage server | |
| US10831386B2 (en) | Remote direct memory access | |
| US10877922B2 (en) | Flushes based on intent log entry states | |
| JP6526235B2 (en) | Data check method and storage system | |
| CN107451013B (en) | Data recovery method, device and system based on distributed system | |
| US20140298078A1 (en) | SYNCHRONOUS MIRRORING OF NVLog TO MULTIPLE DESTINATIONS (ARCHITECTURE LEVEL) | |
| CN107133132B (en) | Data sending method, data receiving method and storage device | |
| CN107038092B (en) | Data copying method and device | |
| CN107122130A (en) | A kind of data delete method and device again | |
| EP3265925A1 (en) | Remote direct memory access | |
| CN107102918B (en) | Data processing method and device | |
| JP2013077074A (en) | Storage device, storage control device, and copy destination data access method | |
| US20190114076A1 (en) | Method and Apparatus for Storing Data in Distributed Block Storage System, and Computer Readable Storage Medium | |
| CN105930545A (en) | Method and device for migrating files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200520 Address after: 215100 No. 1 Guanpu Road, Guoxiang Street, Wuzhong Economic Development Zone, Suzhou City, Jiangsu Province Applicant after: SUZHOU LANGCHAO INTELLIGENT TECHNOLOGY Co.,Ltd. Address before: 450018 Henan province Zheng Dong New District of Zhengzhou City Xinyi Road No. 278 16 floor room 1601 Applicant before: ZHENGZHOU YUNHAI INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |