CN114296941A - Method, system and related device for communication among nodes of distributed cluster system - Google Patents
Method, system and related device for communication among nodes of distributed cluster system Download PDFInfo
- Publication number
- CN114296941A CN114296941A CN202111671748.0A CN202111671748A CN114296941A CN 114296941 A CN114296941 A CN 114296941A CN 202111671748 A CN202111671748 A CN 202111671748A CN 114296941 A CN114296941 A CN 114296941A
- Authority
- CN
- China
- Prior art keywords
- node
- management
- nodes
- data
- common
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000004891 communication Methods 0.000 title claims abstract description 27
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 110
- 230000008569 process Effects 0.000 description 8
- 230000007246 mechanism Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Images
Abstract
The application provides a communication method between nodes of a distributed cluster system, which comprises the following steps: acquiring node reported data of a common node; calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule; determining a corresponding management node according to the main node and the standby node; wherein the number of management nodes is at least 2; and uploading the data reported by the node to the management node. By decentralizing the storage cluster, a series of problems occurring when the main node fails can be avoided, and the reliability of cluster management is improved. The application also provides a communication system among nodes of the distributed cluster system, a computer readable storage medium and a server, and a computer readable storage medium and a server, which have the beneficial effects, and are not described herein again.
Description
Technical Field
The present application relates to the field of server devices, and in particular, to a method, a system, and a related device for communication between nodes in a distributed cluster system.
Background
In a distributed storage cluster, the number of nodes is large, and a management system needs to collect and process various data of each node. In order not to affect the overall performance of the cluster, a general strategy is to select a small number of nodes as management nodes to process the management data, and provide a managed virtual ip inside the cluster, where the collected and reported data of all nodes are sent to the node where the virtual ip is located. The node is called a master node for short, the master node synchronizes the current own data to other management nodes at regular time, and when the master node fails, the virtual ip automatically drifts to other management nodes to become a new master node. This model can solve most of the failure scenarios of the distributed storage cluster, but has the following problems:
1. messages generated by the cluster are lost during the period when the master node is down and a new master node has not been elected.
2. Data synchronization between the master node and the slave node is inefficient, and consumes certain resources, thereby affecting the overall performance of the cluster.
Disclosure of Invention
The purpose of the application is to provide a distributed cluster system inter-node communication method, a distributed cluster system inter-node communication system, a computer readable storage medium and a server, which can change a master-slave structure and improve the stability of a storage cluster.
In order to solve the above technical problem, the present application provides a method for communication between nodes of a distributed cluster system, which has the following specific technical scheme:
acquiring node reported data of a common node;
calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule;
determining a corresponding management node according to the main node and the standby node; wherein the number of management nodes is at least 2;
and uploading the data reported by the node to the management node.
Optionally, the method further includes:
calculating the node reported data volume received by the management node according to the number of common nodes, the number of management nodes and the redundancy rate of the management nodes;
if the number of the nodes is N, the number of the management nodes is N, the redundancy rate of the management nodes is P, and the data volume reported by the nodes received by the management nodes is M/N P.
Optionally, the method further includes:
and if the common node fails, restoring the node data of the common node according to the node reported data stored in the target management node corresponding to the common node.
Optionally, if there is a fault management node, the method further includes:
the redundancy of the remaining management nodes is improved;
and changing the reporting node corresponding to the common node managed by the fault management node into the redundant management node of the management node.
The present application further provides a distributed cluster system inter-node communication system, including:
the data acquisition module is used for acquiring node reported data of the common node;
the node calculation module is used for calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule;
a management node determining module, configured to determine a corresponding management node according to the master node and the standby node; wherein the number of management nodes is at least 2;
and the reporting module is used for uploading the node reported data to the management node.
Optionally, the method further includes:
the data volume calculation module is used for calculating the node reported data volume received by the management node according to the number of common nodes, the number of management nodes and the redundancy of the management nodes;
if the number of the nodes is N, the number of the management nodes is N, the redundancy rate of the management nodes is P, and the data volume reported by the nodes received by the management nodes is M/N P.
Optionally, the method further includes:
and the first fault processing module is used for recovering the node data of the common node according to the node reported data stored by the target management node corresponding to the common node when the common node fails.
Optionally, the method further includes:
the second fault processing module is used for improving the redundancy of the remaining management nodes when the fault management nodes exist; and changing the reporting node corresponding to the common node managed by the fault management node into the redundant management node of the management node.
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method as set forth above.
The present application further provides a server comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described above when calling the computer program in the memory.
The application provides a communication method between nodes of a distributed cluster system, which comprises the following steps: acquiring node reported data of a common node; calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule; determining a corresponding management node according to the main node and the standby node; wherein the number of management nodes is at least 2; and uploading the data reported by the node to the management node.
When the data reported by the nodes are received, the reported data are not uploaded to the corresponding main node or the corresponding standby node, but the corresponding management nodes are calculated, so that the reported data are uploaded to the management nodes, and each management node can process the reported data. By decentralizing the storage cluster, a series of problems occurring when the main node fails can be avoided, thereby improving the reliability of cluster management.
The application also provides a communication system among nodes of the distributed cluster system, a computer readable storage medium and a server, and a computer readable storage medium and a server, which have the beneficial effects, and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for communication between nodes in a distributed cluster system according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a communication system between nodes of a distributed cluster system according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a flowchart of a method for communication between nodes of a distributed cluster system according to an embodiment of the present application, where the method includes:
s101: acquiring node reported data of a common node;
s102: calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule;
s103: determining a corresponding management node according to the main node and the standby node;
s104: and uploading the data reported by the node to the management node.
In step S101, the node reporting data of the common node needs to be acquired, where the common node refers to a non-management node in the storage cluster, that is, a server node that performs a conventional data storage function. In the process of executing step S101, a conventional inter-node communication method may be adopted, which is not specifically limited herein.
In step S102, the main node and the standby node corresponding to the data to be reported by the computing node are required. However, it should be noted that, at this time, there is no specific limitation on whether the primary node and the standby node exist in the storage cluster. On one hand, the method and the device can be applied to the storage cluster with the existing main node and standby node, so that the management node is directly configured. On the other hand, the method and the device can also cancel the election mechanism of the main node aiming at the storage cluster with the main node and the standby node, so that only the management node is configured. Of course, for a storage cluster without a master/slave node election mechanism, a management node may be configured directly as in the present application.
In step S102, the master node and the standby node to which the data reported by the node should be uploaded need to be calculated according to a preset hash algorithm rule. Since a redundancy mechanism is usually configured in the storage cluster, that is, the same data is generally stored in multiple nodes at the same time, the data reported by the node usually needs to be uploaded to at least two nodes. The preset hash algorithm rule used is not limited, and different preset hash algorithm rules may be configured with reference to an upload path, upload efficiency, upload cost, or the like, so as to determine a master node and a standby node to which data reported by the node should be uploaded.
Thereafter, in step S103, a corresponding management node is determined from the primary node and the backup node. First, at least one management node is determined in this step, and two or more management nodes may be determined. If the number of the determined management nodes is one, the management node is used as a first storage node of the data reported by the node, and when the common node is recovered by using the data reported by the node, the management node is selected as the highest priority. Furthermore, it is also possible to determine a data redundancy mechanism between a plurality of management nodes, i.e. configuration management nodes. For example, if there are 100 common nodes, 5 management nodes are configured, and each management node should receive the data reported by 20 common nodes on average, and if a backup mechanism is configured, each management node should receive the data of 40 nodes on average. Therefore, when any management node fails, the loss of management data cannot be influenced.
On the basis of this embodiment, the node reported data amount received by the management node, that is, the management node redundancy, may be calculated according to the number of common nodes, the number of management nodes, and the management node redundancy, where the unit management node bears redundant data amount, and the greater the management node redundancy, the more redundant data is loaded by each management node. Redundant data refers to data that is already stored on other management nodes. For example, if a management node redundancy of 2 indicates that two copies are stored per management node, a management node redundancy of 3 indicates that three copies are stored per management node. When the data volume reported by the node acceptable by each management node is calculated, the following calculation mode can be adopted:
and if the number of the nodes is N, the number of the management nodes is N, the redundancy of the management nodes is P, and the data volume reported by the nodes received by the management nodes is M/N P. Of course, those skilled in the art may also configure the node reporting data amount based on other calculation manners, and the scope of the present application should be also within the scope of the present application.
On the basis of the implementation, if the common node fails, the node data of the common node can be recovered according to the node reported data stored by the target management node corresponding to the common node.
Similarly, if a fault management node exists, the redundancy of the remaining management nodes can be improved, and meanwhile, the reporting node corresponding to the common node managed by the fault management node is changed to be the redundancy management node of the management node. That is, the common node that originally needs to be uploaded to the fault management node is changed to be reported to other management nodes, and the redundancy of each management node should be properly improved due to the reduction of the number of the management nodes. If more management node faults exist at the same time, a new management node can be set.
When the data reported by the nodes are received, the reported data are not uploaded to the corresponding main node or the corresponding standby node, but the corresponding management nodes are calculated, so that the reported data are uploaded to the management nodes, and each management node can process the reported data. By decentralizing the storage cluster, a series of problems occurring when the main node fails can be avoided, thereby improving the reliability of cluster management.
In the following, a communication system between nodes of a distributed cluster system provided in an embodiment of the present application is introduced, and a communication system between nodes of a distributed cluster system described below and a communication method between nodes of a distributed cluster system described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a communication system between nodes of a distributed cluster system according to an embodiment of the present application, and the present application further provides a communication system between nodes of a distributed cluster system, including:
the data acquisition module is used for acquiring node reported data of the common node;
the node calculation module is used for calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule;
a management node determining module, configured to determine a corresponding management node according to the master node and the standby node; wherein the number of management nodes is at least 2;
and the reporting module is used for uploading the node reported data to the management node.
Based on the above embodiment, as a more preferred embodiment, the method further includes:
the data volume calculation module is used for calculating the node reported data volume received by the management node according to the number of common nodes, the number of management nodes and the redundancy of the management nodes;
if the number of the nodes is N, the number of the management nodes is N, the redundancy rate of the management nodes is P, and the data volume reported by the nodes received by the management nodes is M/N P.
Based on the above embodiment, as a more preferred embodiment, the method further includes:
and the first fault processing module is used for recovering the node data of the common node according to the node reported data stored by the target management node corresponding to the common node when the common node fails.
Based on the above embodiment, as a more preferred embodiment, the method further includes:
the second fault processing module is used for improving the redundancy of the remaining management nodes when the fault management nodes exist; and changing the reporting node corresponding to the common node managed by the fault management node into the redundant management node of the management node.
The present application also provides a computer readable storage medium having stored thereon a computer program which, when executed, may implement the steps provided by the above-described embodiments. The storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The application also provides a server, which may include a memory and a processor, where the memory stores a computer program, and the processor may implement the steps provided by the foregoing embodiments when calling the computer program in the memory. Of course, the server may also include various network interfaces, power supplies, and the like.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system provided by the embodiment, the description is relatively simple because the system corresponds to the method provided by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
It is further noted that, in the present specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for communication among nodes of a distributed cluster system is characterized by comprising the following steps:
acquiring node reported data of a common node;
calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule;
determining a corresponding management node according to the main node and the standby node; wherein the number of management nodes is at least 2;
and uploading the data reported by the node to the management node.
2. The method of claim 1, further comprising:
calculating the node reported data volume received by the management node according to the number of common nodes, the number of management nodes and the redundancy rate of the management nodes;
if the number of the nodes is N, the number of the management nodes is N, the redundancy rate of the management nodes is P, and the data volume reported by the nodes received by the management nodes is M/N P.
3. The method of claim 1, further comprising:
and if the common node fails, restoring the node data of the common node according to the node reported data stored in the target management node corresponding to the common node.
4. The method of claim 1, wherein if there is a fault management node, further comprising:
the redundancy of the remaining management nodes is improved;
and changing the reporting node corresponding to the common node managed by the fault management node into the redundant management node of the management node.
5. A distributed inter-node communication system for a cluster system, comprising:
the data acquisition module is used for acquiring node reported data of the common node;
the node calculation module is used for calculating a main node and a standby node corresponding to the node reported data according to a preset Hash algorithm rule;
a management node determining module, configured to determine a corresponding management node according to the master node and the standby node; wherein the number of management nodes is at least 2;
and the reporting module is used for uploading the node reported data to the management node.
6. The distributed cluster system inter-node communication system of claim 5, further comprising:
the data volume calculation module is used for calculating the node reported data volume received by the management node according to the number of common nodes, the number of management nodes and the redundancy of the management nodes;
if the number of the nodes is N, the number of the management nodes is N, the redundancy rate of the management nodes is P, and the data volume reported by the nodes received by the management nodes is M/N P.
7. The distributed cluster system inter-node communication system of claim 5, further comprising:
and the first fault processing module is used for recovering the node data of the common node according to the node reported data stored by the target management node corresponding to the common node when the common node fails.
8. The distributed cluster system inter-node communication system of claim 5, further comprising:
the second fault processing module is used for improving the redundancy of the remaining management nodes when the fault management nodes exist; and changing the reporting node corresponding to the common node managed by the fault management node into the redundant management node of the management node.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the distributed cluster system inter-node communication method according to any one of claims 1 to 4.
10. A server, comprising a memory having a computer program stored therein and a processor that when invoked on the computer program in the memory performs the steps of the distributed cluster system inter-node communication method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111671748.0A CN114296941A (en) | 2021-12-31 | 2021-12-31 | Method, system and related device for communication among nodes of distributed cluster system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111671748.0A CN114296941A (en) | 2021-12-31 | 2021-12-31 | Method, system and related device for communication among nodes of distributed cluster system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114296941A true CN114296941A (en) | 2022-04-08 |
Family
ID=80975402
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111671748.0A Withdrawn CN114296941A (en) | 2021-12-31 | 2021-12-31 | Method, system and related device for communication among nodes of distributed cluster system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114296941A (en) |
-
2021
- 2021-12-31 CN CN202111671748.0A patent/CN114296941A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3694148B1 (en) | Configuration modification method for storage cluster, storage cluster and computer system | |
| US8874960B1 (en) | Preferred master election | |
| CN112463448B (en) | Distributed cluster database synchronization method, device, equipment and storage medium | |
| EP1550036A2 (en) | Method of solving a split-brain condition | |
| TWI701916B (en) | Method and device for self-recovering management ability in distributed system | |
| CN112422320B (en) | Master-slave switching method and device of server and server | |
| CN101136728A (en) | Cluster system and method for backing up a replica in a cluster system | |
| CN107360025B (en) | Distributed storage system cluster monitoring method and device | |
| JP4491482B2 (en) | Failure recovery method, computer, cluster system, management computer, and failure recovery program | |
| CN109165112B (en) | Fault recovery method, system and related components of metadata cluster | |
| CN104468651A (en) | Distributed multi-copy storage method and device | |
| CN112130758A (en) | Data reading request processing method and system, electronic equipment and storage medium | |
| CN110635941A (en) | Database node cluster fault migration method and device | |
| CN110377664B (en) | Data synchronization method, device, server and storage medium | |
| CN107508700B (en) | Disaster recovery method, device, equipment and storage medium | |
| CN117061512A (en) | Distributed information management method and system based on big data | |
| CN112130761A (en) | Capacity expansion method and system for storage system and related device | |
| US20210096962A1 (en) | Data backup method, device and system | |
| CN114296941A (en) | Method, system and related device for communication among nodes of distributed cluster system | |
| CN114697353B (en) | Distributed storage cluster power grid data storage control method | |
| CN111309515A (en) | Disaster recovery control method, device and system | |
| CN115756955A (en) | Data backup and data recovery method and device and computer equipment | |
| CN111817892B (en) | Network management method, system, electronic equipment and storage medium | |
| CN111324513B (en) | Monitoring management method and system for artificial intelligence development platform | |
| CN108897645B (en) | Database cluster disaster tolerance method and system based on standby heartbeat disk |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20220408 |
|
| WW01 | Invention patent application withdrawn after publication |