CN112311596B

CN112311596B - Data management method, device, equipment and computer storage medium

Info

Publication number: CN112311596B
Application number: CN202011139659.7A
Authority: CN
Inventors: 焦宏宇; 邱路达; 丁易元; 张若君; 孙芮; 邓虹雨
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2020-10-22
Filing date: 2020-10-22
Publication date: 2023-05-12
Anticipated expiration: 2040-10-22
Also published as: WO2022083690A1; CN112311596A

Abstract

The embodiment of the application provides a data management method, a data management device, electronic equipment and a computer storage medium; the method comprises the following steps: establishing a data storage architecture, wherein the data storage architecture comprises a parent cluster and a plurality of sub-clusters, each sub-cluster in the plurality of sub-clusters comprises a master node and at least one slave node, the nodes of the parent cluster are the master nodes of the plurality of sub-clusters, and the nodes of the parent cluster comprise the master node of the parent cluster and at least one slave node of the parent cluster; based on the data storage architecture, data is written to each node of each sub-cluster.

Description

Data management method, device, equipment and computer storage medium

Technical Field

The present application relates to the field of distributed architecture technology of financial technology (Fintech), and relates to, but is not limited to, a data management method, apparatus, electronic device, and computer storage medium.

Background

With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changed into financial technology, but due to the requirements of safety and instantaneity of the financial industry, higher requirements are also put on the technologies.

At present, in the field of financial science and technology, data management can be realized based on a single-layer distributed architecture, and the single-layer distributed architecture is not suitable for oversized distributed clusters, and specifically because oversized distributed cluster nodes are numerous, the premise of carrying out data management based on oversized distributed clusters is as follows: when cluster election is carried out based on a single-layer distributed architecture, however, each node sends election messages to other nodes in the cluster, so that message storm is easily caused, most traffic is occupied by messages sent by the cluster election, and normal message transmission is affected.

Disclosure of Invention

The embodiment of the application provides a data management method, a data management device, electronic equipment and a computer storage medium, which can solve the problem of message storm caused by electing a master node in the prior art.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a data management method, which comprises the following steps:

establishing a data storage architecture, wherein the data storage architecture comprises a Parent Cluster (Parent Cluster) and a plurality of sub-clusters (Child clusters), each sub-Cluster comprises a master node and at least one slave node, the nodes of the Parent Cluster are the master nodes of the sub-clusters, and the nodes of the Parent Cluster comprise the master node of the Parent Cluster and the at least one slave node of the Parent Cluster;

Based on the data storage architecture, data is written to each node of each sub-cluster.

In some embodiments of the present application, the establishing a data storage architecture includes:

determining main nodes and health degree indexes of all sub-clusters;

based on the health degree index of each sub-cluster, selecting a main node from the main nodes of each sub-cluster as the main node of the father cluster;

the data storage architecture is established based on the master nodes of the respective sub-clusters and the master nodes of the parent clusters.

In some embodiments of the present application, the method further comprises:

deriving a health indicator for the master node of each sub-cluster based on at least one of: the number of faults, the duration of at least one fault, the duration of the last fault from the current time.

It can be seen that the embodiment of the application can accurately obtain the health degree index of the main node of each sub-cluster based on the fault information, thereby being beneficial to accurately selecting the node with higher health degree from the main nodes of the sub-clusters as the main node of the father cluster,

in some embodiments of the present application, the selecting, based on the health index of each of the sub-clusters, one master node from among the master nodes of each of the sub-clusters as the master node of the parent cluster includes:

Selecting a main node from the main nodes of each sub-cluster as the main node of the father cluster based on the health index and the first grading value of the main node of each sub-cluster; wherein the first scoring value is a preconfigured value.

It can be seen that in the embodiment of the present application, the selection of the master node of the parent cluster is not only performed based on the ID of the node, but the master node of the parent cluster can be selected on the basis of comprehensively considering the score and the health index corresponding to the ID of the node, which is favorable for reducing the problem of frequent replacement of the master node caused by frequent addition of the node with a larger score of the ID into the cluster.

In some embodiments of the present application, the selecting, based on the health index and the first score value of each of the sub-clusters, one master node from among the master nodes of each of the sub-clusters as the master node of the parent cluster includes:

based on the data interaction of the main nodes of each sub-cluster, the main node of each sub-cluster obtains a second grading value of the main node of each sub-cluster, wherein the second grading value is the sum of the health degree index and the first grading value;

and based on the second grading value of the main nodes of the sub-clusters, the main nodes of the father cluster are elected by the main nodes of the sub-clusters.

It can be appreciated that, according to the embodiment of the present application, the primary node of the parent cluster may be obtained through election of the child cluster based on the second score value, where the second score value is related to not only the ID of the node but also the health index of the node, so that the problem that the primary node is frequently replaced due to frequent joining of the node with a larger ID score into the cluster can be reduced.

based on the data interaction of the main nodes of the sub-clusters, the main node of each sub-cluster acquires a first Message of each sub-cluster, wherein the first Message represents a Message (Join Message) of joining a parent cluster;

and under the condition that the master node of each sub-cluster acquires the first message of each sub-cluster, selecting one master node from the master nodes of each sub-cluster as the master node of the father cluster based on the health degree index of each sub-cluster.

It can be seen that, according to the embodiment of the present application, based on the first message of the master node of each sub-cluster, a process of selecting the master node of the parent cluster is initiated at a suitable time, which is favorable for accurately selecting the master node of the parent cluster from the master nodes of each sub-cluster.

In some embodiments of the present application, the data interaction based on the master node of each sub-cluster, the enabling the master node of each sub-cluster to obtain the first message of each sub-cluster includes:

selecting a Seed Cluster (Seed Cluster) from the respective sub-clusters;

after receiving the first message sent by the master node of the other sub-clusters, the master node (Seed Cluster Master) of the seed cluster makes the master node of each sub-cluster acquire the first message of each sub-cluster through the data interaction between the seed cluster and the other sub-clusters; wherein the other sub-clusters are sub-clusters of the plurality of sub-clusters other than the seed cluster; the first message is sent by the master node of the other sub-cluster based on the address of the master node of the seed cluster, which is information predetermined by the master node of the other sub-cluster.

It can be seen that, according to the embodiment of the application, the data interaction of the main nodes of each sub-cluster can be realized through the selection of the seed cluster, and further, the selection of the main node of the father cluster from the main nodes of each sub-cluster is facilitated.

In some embodiments of the present application, the method further comprises:

Deleting the failed slave node from the parent cluster under the condition that the slave node of the parent cluster fails, and transmitting member change information of the parent cluster to the slave node of the parent cluster by utilizing the master node of the parent cluster;

after receiving a first message sent by a master node of a first sub-cluster, a master node of the first sub-cluster is added into the parent cluster, and member change information of the parent cluster is sent to each node of the parent cluster; the first sub-cluster represents a sub-cluster to which the failed slave node belongs, and the master node of the first sub-cluster represents a master node reselected from all the nodes of the first sub-cluster; the first message represents a message joining a parent cluster.

It can be understood that, in the embodiment of the present application, when the slave node of the parent cluster fails, the member change information of the parent cluster may be updated in time, and when the master node of the parent cluster is selected subsequently, the master node of the parent cluster may be accurately selected.

In some embodiments of the present application, the method further comprises:

under the condition that the main node of the father cluster fails, selecting one node from other nodes of the father cluster as the main node of the father cluster;

After receiving a first message sent by a main node of a second sub-cluster, adding the main node of the second sub-cluster into the father cluster, and sending member change information of the father cluster to each node of the father cluster; the second subset group represents a subset group to which the failed master node belongs, and the first message represents a message joining the parent group.

It can be seen that when the master node of the parent cluster fails, the master node information of the parent cluster can be reelected in time, and after the master node is selected from the second subset cluster, the joining message of the master node of the second subset cluster can be received, so that the reelect of the master node of the subsequent parent cluster is facilitated.

In some embodiments of the present application, the method further comprises:

acquiring a data reading instruction, wherein the data reading instruction carries a data reading address of any node;

and reading data from any one node based on the data reading address.

It can be seen that, because the data in each sub-cluster is consistent, the embodiment of the application can read the data based on any node in any sub-cluster, and has the characteristic of easy implementation.

In some embodiments of the present application, the writing data to each node of each sub-cluster based on the data storage architecture includes:

acquiring a data writing instruction, wherein the data writing instruction carries data to be written;

and sending the data writing instruction to the master node of the father cluster, sending the data writing instruction to the master nodes of all the sub-clusters by the master node of each sub-cluster, and sending the data writing instruction to each slave node of the sub-cluster by the master node of each sub-cluster, so that each node of each sub-cluster writes the data to be written.

It can be seen that the embodiment of the application can write the same data to each node of each sub-cluster based on the data writing instruction, so that the data consistency of each sub-cluster can be realized.

In some embodiments of the present application, the data to be written includes at least two levels of tag data;

writing the data to be written to each node of each sub-cluster comprises:

establishing a hash table for each level of tag data, and constructing a multi-way tree data structure based on the hash table corresponding to each level of tag data;

the at least two levels of tag data are written into each node of each sub-cluster based on the multi-drop data structure.

It can be seen that the embodiment of the application can realize hierarchical storage of the multi-level tag data based on the multi-tree data structure, and is beneficial to subsequent management of the multi-level tag data.

In some embodiments of the present application, in a case where any one level of tag data is not the lowest level of tag data, the hash table of any one level of tag data includes a hash address corresponding to the next level of tag data.

It can be seen that, based on the multi-tree data structure in the embodiment of the present application, the tag data of the current level can be quickly queried to the tag data of the next level.

In some embodiments of the present application, the method further comprises:

based on the multi-way tree data structure, performing at least one of the following on the at least two levels of tag data: add, delete, modify, query.

It can be seen that the method is favorable for rapidly realizing the addition, deletion, modification or inquiry of at least two levels of tag data based on the characteristics of the multi-tree data structure.

An embodiment of the present application provides a data management apparatus, including:

the system comprises a building module, a data storage architecture, a data storage module and a data storage module, wherein the data storage architecture comprises a father cluster and a plurality of sub-clusters, each sub-cluster in the plurality of sub-clusters comprises a master node and at least one slave node, the nodes of the father cluster are the master nodes of the plurality of sub-clusters, and the nodes of the father cluster comprise the master node of the father cluster and at least one slave node of the father cluster;

And the processing module is used for writing data into each node of each sub-cluster based on the data storage architecture.

In some embodiments of the present application, the establishing module, configured to establish a data storage architecture, includes:

determining main nodes and health degree indexes of all sub-clusters;

In some embodiments of the present application, the establishing module is further configured to derive the health indicator of the master node of each sub-cluster according to at least one of: the number of faults, the duration of at least one fault, the duration of the last fault from the current time.

In some embodiments of the present application, the establishing module is configured to select, from the master nodes of the respective sub-clusters, a master node as the master node of the parent cluster based on the health indicator of the respective sub-clusters, including:

In some embodiments of the present application, the establishing module is configured to select, from the master nodes of the respective sub-clusters, a master node as the master node of the parent cluster based on the health indicator and the first score value of the respective sub-clusters, including:

based on the data interaction of the main nodes of each sub-cluster, the main node of each sub-cluster acquires a first message of each sub-cluster, wherein the first message represents a message added into a parent cluster;

In some embodiments of the present application, the establishing module is configured to, based on data interaction of the master node of each sub-cluster, cause the master node of each sub-cluster to obtain a first message of each sub-cluster, including:

selecting seed clusters from the sub-clusters;

after the main node of the seed cluster receives the first messages sent by the main nodes of other sub-clusters, the main node of each sub-cluster acquires the first messages of the sub-clusters through the data interaction between the seed cluster and the other sub-clusters; wherein the other sub-clusters are sub-clusters of the plurality of sub-clusters other than the seed cluster; the first message is sent by the master node of the other sub-cluster based on the address of the master node of the seed cluster, which is information predetermined by the master node of the other sub-cluster.

In some embodiments of the present application, the establishing module is further configured to:

After receiving a first message sent by a master node of a first sub-cluster, a master node of the first sub-cluster is added into the parent cluster, and member change information of the parent cluster is sent to each node of the parent cluster; the first sub-cluster represents a sub-cluster to which the failed slave node belongs, the master node of the first sub-cluster represents a master node reselected from each node of the first sub-cluster, and the first message represents a message joining a parent cluster.

In some embodiments of the present application, the processing module is further configured to:

and reading data from any one node based on the data reading address.

In some embodiments of the present application, the processing module, configured to write data to each node of each sub-cluster based on the data storage architecture, includes:

writing the data to be written to each node of each sub-cluster comprises:

In some embodiments of the present application, the processing module is further configured to perform, based on the multi-tree data structure, at least one of the following operations on the at least two levels of tag data: add, delete, modify, query.

An embodiment of the present application provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for realizing any one of the data management methods when executing the executable instructions stored in the memory.

The embodiment of the application provides a computer readable storage medium, which stores executable instructions for implementing any one of the above data management methods when executed by a processor.

In the embodiment of the application, a data storage architecture is established, wherein the data storage architecture comprises a father cluster and a plurality of sub-clusters, each sub-cluster in the plurality of sub-clusters comprises a master node and at least one slave node, the nodes of the father cluster are the master nodes of the plurality of sub-clusters, and the nodes of the father cluster comprise the master node of the father cluster and at least one slave node of the father cluster; based on the data storage architecture, data is written to each node of each sub-cluster. It can be seen that the data storage architecture of the embodiment of the application is a dual-layer distributed architecture, and can realize data interaction based on the main nodes of each sub-cluster, when cluster election is performed, all sub-nodes are not required to send elected negotiation messages, negotiation message data is reduced, the propagation range of the negotiation messages is limited, and the problem that normal message transmission is influenced due to message storm is facilitated to be alleviated.

Drawings

FIG. 1 is an alternative flow chart of a data management method provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a data storage architecture of an embodiment of the present application;

FIG. 3 is a schematic diagram of cluster set-up in an embodiment of the present application;

FIG. 4 is a schematic diagram of an electing a parent cluster master node according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a multi-drop tree data structure according to an embodiment of the present application;

FIG. 6 is a flow chart of performing an add data operation on at least two levels of tag data in an embodiment of the present application;

FIG. 7 is a flow chart of a query data operation performed on at least two levels of tag data in an embodiment of the present application;

FIG. 8 is a schematic view of an alternative composition of a data management device according to an embodiment of the present application;

fig. 9 is a schematic diagram of an alternative composition structure of an electronic device according to an embodiment of the present application.

Detailed Description

In the related art, data management can be implemented based on a single-layer distributed architecture, which has the following drawbacks:

1) The single-layer distributed architecture is not suitable for ultra-large distributed clusters, because the ultra-large distributed clusters have a plurality of nodes, and the precondition for data management based on the ultra-large distributed clusters is as follows: the master node is determined through cluster election, however, when cluster election is carried out, each node transmits election information to other nodes in the cluster, so that message storm is easy to cause, most of traffic is occupied by messages transmitted by the cluster election, and normal message transmission is affected.

2) When the main node is determined through cluster election, the election algorithm does not balance the election speed and stability, for example, the trunk algorithm is simply judged according to the size of the node ID when the cluster election is carried out, which causes the problem of frequent election of the main node; for example, a shift algorithm is used for cluster election, and a majority voting mechanism is used, which can avoid frequent election, but can lead to longer election time, and more than half nodes must vote to elect a master node.

3) When a single-layer distributed architecture is used for realizing data storage, each node uses a unified Hash (Hash) table structure for data storage, and when more stored data needs to be expanded, hash values of all key values (keys) involved in the nodes need to be calculated, so that the expansion time is prolonged.

Aiming at the technical problems, the technical scheme of the embodiment of the application is provided.

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings, and the described embodiments should not be construed as limiting the present application, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present application.

The embodiment of the application provides a data management method, a device, an electronic device and a computer storage medium, where the data management method of the embodiment of the application may be applied to the electronic device, and an exemplary application of the electronic device provided by the embodiment of the application is described below, where the electronic device provided by the embodiment of the application may be implemented as a server, and the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), big data and artificial intelligent platforms, and the like.

The following describes an exemplary data management method according to an embodiment of the present application.

Fig. 1 is an optional flowchart of a data management method according to an embodiment of the present application, where, as shown in fig. 1, the flowchart may include:

step 101: the method comprises the steps of establishing a data storage architecture, wherein the data storage architecture comprises a parent cluster and a plurality of sub-clusters, each sub-cluster in the plurality of sub-clusters comprises a master node and at least one slave node, the nodes of the parent cluster are master nodes of the plurality of sub-clusters, and the nodes of the parent cluster comprise a master node of the parent cluster and at least one slave node of the parent cluster.

In the embodiment of the application, the sub-clusters represent small clusters obtained by dividing nodes of all clusters, each sub-cluster can independently elect a sub-cluster master node (Child Cluster Master), and other nodes except the sub-cluster master node in the sub-cluster are sub-cluster slave nodes (Child Cluster Slave).

The parent cluster represents a cluster composed of the subset group master nodes, one node can be selected from the nodes of the parent cluster as a parent group master node (Parent Cluster Master), and the rest nodes except the parent group master node in the parent cluster are parent group slave nodes (Parent Cluster Slave).

FIG. 2 is a schematic diagram of a data storage architecture according to an embodiment of the present application, wherein, as shown in FIG. 2, subset group 0, subset group 1 and subset group 2 represent 3 different subset groups; master node 01 represents the master node of sub-cluster 0, and slave node 02 and slave node 03 represent different slave nodes in sub-cluster 0; the master node 11 represents the master node of the sub-cluster 1, and the

slave nodes

12 and 13 represent different slave nodes in the sub-cluster 1; master node 21 represents the master node of sub-cluster 2, and

slave nodes

22 and 23 represent different slave nodes in sub-cluster 2; the master node 01, the master node 11 and the master node 21 form a parent cluster, the master node 21 is a parent cluster master node, and the master node 11 and the master node 01 are different slave nodes in the parent cluster.

Referring to fig. 2, the data storage architecture of the embodiment of the present application is a dual-layer structure, where a first layer is formed by a plurality of sub-clusters, and the sub-clusters of the first layer include sub-cluster 0, sub-cluster 1 and sub-cluster 2; the second layer represents a parent cluster consisting of the master nodes of the plurality of child clusters, i.e. the parent cluster consists of master node 01, master node 11 and master node 21. After the establishment of each sub-cluster is completed, the respective main node can be independently elected, and then the main grounding of the parent cluster can be obtained through the election of the main nodes of each sub-cluster, so that each super-large cluster is realized through a double-layer structure.

Step 102: based on the data storage architecture, data is written to each node of each sub-cluster.

In the embodiment of the present application, based on the data storage structure, after receiving the data writing instruction, the same data may be written to each node of each sub-cluster based on the data writing instruction, so that the data consistency of each sub-cluster may be achieved. Based on the data storage structure described above, data can be read from any one node because the data in the respective nodes is consistent.

In practical applications, steps 101 to 102 may be implemented based on a processor of an electronic device, where the processor may be at least one of an application specific integrated circuit (Application Specific Integrated Circuit, asic), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic Device, PLD), a field programmable gate array (Field Programmable Gate Array, fpga), a CPU, a controller, a microcontroller, and a microprocessor. It will be appreciated that the electronic device implementing the above-described processor function may be other, and embodiments of the present application are not limited.

It can be seen that the data storage architecture of the embodiment of the application is a dual-layer distributed architecture, and can realize data interaction based on the main nodes of each sub-cluster, when cluster election is performed, all sub-nodes are not required to send elected negotiation messages, negotiation message data is reduced, the propagation range of the negotiation messages is limited, and the problem that normal message transmission is influenced due to message storm is facilitated to be alleviated.

In some embodiments of the present application, establishing a data storage architecture may include: determining main nodes and health degree indexes of all sub-clusters; based on the health degree index of each sub-cluster, selecting a main node from the main nodes of each sub-cluster as the main node of the father cluster; the data storage architecture is established based on the master nodes of the respective sub-clusters and the master nodes of the parent clusters.

Here, the health index is used to reflect the health of the master node of the sub-cluster; after determining the master node of each sub-cluster, the slave node of each sub-cluster can be determined, and after determining the master node of the parent cluster, the slave node of the parent cluster can be determined, so that the data storage architecture can be obtained.

It can be understood that, unlike the scheme of selecting a master node only according to the ID of the node in the related art, the embodiment of the present disclosure may select a child node of a parent cluster based on the health index of the master node of each child cluster, and compared with the buly algorithm, even if a node with a larger ID is frequently added, frequent replacement of the master node of the parent cluster is not caused, so that the stability is stronger; compared with the raft algorithm, the method has the advantages that the election of the main node is required to be carried out based on a half voting mechanism, the speed of determining the main node is higher, and the balance of the election speed and stability is realized to a certain extent.

In some embodiments of the present application, the health indicator of the master node of each sub-cluster may be derived from at least one of: the number of faults, the duration of at least one fault, the duration of the last fault from the current time.

Here, the health degree index is inversely related to the number of times of occurrence of the fault, the health degree index is inversely related to the duration of at least one fault, and the health degree index is distant from the duration of the current time from the last fault; that is, the shorter the duration of at least one fault, the greater the value of the health indicator; the longer the last fault is from the current time, the larger the numerical value of the health degree index is; the greater the number of failures, the smaller the value that indicates the health index.

In some embodiments, the duration of the at least one fault may include the duration of the last fault.

In practical applications, the failure start time and the failure recovery time may be recorded at each failure of the node, so as to determine the failure duration.

In some embodiments, the health indicator for the master node of each sub-cluster may be derived according to the following equation (1):

S ₀ ＝t ₁ -(fault_count *t ₂ ) (1)

wherein S is ₀ Health index, t, representing the master node of a sub-cluster ₁ The time length of the last fault from the current time is represented, fault_count represents the number of times of faults, t ₂ Indicating the duration of the last failure.

It can be understood that the embodiment of the application can accurately obtain the health degree index of the main node of each sub-cluster based on the fault information, thereby being beneficial to accurately selecting the node with higher health degree from the main nodes of the sub-clusters as the main node of the father cluster,

in some embodiments of the present application, one master node may be selected as a master node of a parent cluster from the master nodes of each sub-cluster based on the health index and the first score value of the master node of each sub-cluster; wherein the first scoring value is a preconfigured value.

In some embodiments, the first scoring value may be a score corresponding to an ID of the node; in practical applications, the first score value may be predetermined, i.e. the first score value of each node is a fixed constant value.

In some embodiments of the present application, the selecting, based on the health index and the first score value of each sub-cluster, one master node from the master nodes of each sub-cluster as the master node of the parent cluster may include:

In practical application, the master node of each sub-cluster can calculate the second score value of the node, and then through the data interaction of the master nodes of each sub-cluster, the second score values of the master nodes of other sub-clusters can be obtained, and then the size relationship between the second score values of the node and the master nodes of other sub-clusters can be judged, and whether the node can be used as the master node of the father cluster is judged.

In some embodiments, if the second score value of the present node is greater than or equal to the second score values of the master nodes of other child clusters, then the present node may be said to be the master node of the parent cluster; if the second score value of the node is smaller than the second score value of the master node of any other child cluster, the node cannot be used as the master node of the father cluster.

In some embodiments of the present application, the selecting, based on the health indicator of each sub-cluster, one master node from the master nodes of each sub-cluster as the master node of the parent cluster may include:

based on the data interaction of the main nodes of each sub-cluster, the main node of each sub-cluster acquires a first message of each sub-cluster, wherein the first message represents a message of joining a (join) parent cluster;

under the condition that the main node of each sub-cluster obtains the first message of each sub-cluster, one main node is selected as the main node of the father cluster from the main nodes of each sub-cluster based on the health degree index of each sub-cluster.

In practical application, after generating a first message, namely a message of joining a parent cluster, the master node of each child cluster can send the first message to the master nodes of other child clusters; the master node of each sub-cluster may also receive the first message sent by the master nodes of other sub-clusters; after the primary node of each sub-cluster obtains the first message of each sub-cluster, a process of selecting the primary node of the parent cluster can be initiated.

In some embodiments of the present application, the foregoing data interaction based on the master node of each sub-cluster, making the master node of each sub-cluster obtain the first message of each sub-cluster includes:

selecting seed clusters from the sub-clusters;

after the main node of the seed cluster receives the first messages sent by the main nodes of other sub-clusters, the main node of each sub-cluster acquires the first messages of each sub-cluster through the data interaction between the seed cluster and the other sub-clusters; wherein the other sub-clusters are sub-clusters other than the seed clusters among the plurality of sub-clusters; the first message is sent by the master node of the other sub-cluster based on the address of the master node of the seed cluster, which is predetermined information by the master nodes of the other sub-clusters.

In some embodiments, one sub-cluster may be arbitrarily selected from each sub-cluster as a seed cluster, or a seed cluster may be selected from each sub-cluster according to a preset selection manner.

In some embodiments, the address of the master node of the seed cluster may be preconfigured in the master nodes of the other sub-clusters, such that the master nodes of the other sub-clusters may send the first message to the seed cluster; after receiving the first message, the seed cluster may record the sending address of the first message, that is, the addresses of the main nodes of other sub-clusters, and then, the seed cluster may send the received first message to the main nodes of each sub-cluster.

Fig. 3 is a schematic diagram of cluster establishment in the embodiment of the present application, as shown in fig. 3, a seed cluster is selected first, where the seed cluster is one of all sub-clusters, and the seed cluster, the sub-cluster 1, and the sub-cluster 2 represent 3 different sub-clusters; master node 01 represents the master node of the seed cluster, and

slave nodes

02 and 03 represent slave nodes (Seed Cluster Slave) in the seed cluster; the meanings of the master node 11, the slave node 12 and the slave node 13 are the same as those of fig. 2, and the meanings of the master node 21, the slave node 22 and the slave node 23 are the same as those of fig. 2, and are not repeated here.

The process of cluster set-up is described below with reference to fig. 3.

After the seed cluster is selected, the master node of the seed cluster can be selected, and then the IDs of all the sub-clusters (including the sub-cluster 1 and the sub-cluster 2) at the initial moment can be written into the master node of the seed cluster, so that the seed cluster can obtain the IDs of all the sub-clusters.

Other sub-clusters besides the seed cluster can select a main node, and the ID and the address of the seed cluster can be configured in other sub-clusters in advance.

After each sub-cluster has selected the master node, the master node of each sub-cluster may send a first message to the master node of the seed cluster (e.g., master node 01 shown in fig. 3), where the first message includes information such as an ID of the sub-cluster and an address of the master node of the sub-cluster.

After receiving the first message sent by the other sub-clusters, the master node of the seed cluster can store the ID of the sub-cluster as sub-cluster information according to the first message, and record the address of the master node of the corresponding sub-cluster.

After the seed cluster receives the first messages of all other sub-clusters, the seed cluster can broadcast the sub-cluster information to the main nodes of all other sub-clusters, wherein the sub-cluster information can comprise the received first messages of all other sub-clusters, so that each sub-cluster can acquire the information of the main nodes of all sub-clusters, and at the moment, the election process of the main nodes of the father cluster can be initiated; here, in determining the election process of the master node that initiates the parent cluster, the seed cluster has no meaning and needs to fairly compete with other child clusters.

Here, the seed cluster is a coordinator at the initialization time of the cluster establishment, and each sub-cluster can acquire the information of the master node of each sub-cluster and then degenerate into the sub-clusters.

FIG. 4 is a schematic diagram of a parent cluster master node selected in an embodiment of the present application, where in FIG. 4, child cluster 0, child cluster 1, and child cluster 2 represent 3 different child clusters; the meanings of the master node 01, the slave node 02, the slave node 03, the master node 11, the slave node 12, the slave node 13, the master node 21, the slave node 22 and the slave node 23 are the same as those of fig. 2, and are not described again here.

Referring to fig. 4, the master node of each sub-cluster first determines a second score value of the node and may send a success (vector) message to the master nodes of the other sub-clusters, e.g., master node 21 may send a vector message to master node 11 and master node 01; the vector message indicates a message that declares the node to be a master node, and the vector message may carry the second score value of the node.

The main node of each sub-cluster can receive the victorial information sent by the main nodes of other sub-clusters, after receiving the victorial information, the second grading value in the victorial information is compared with the second grading value of the main node, if the second grading value in the victorial information is greater than or equal to the second grading value of the main node, an election (selection) information can be sent to the source node of the victorial information, and the selection information represents the information of selecting the source node of the victorial information as the main node of the father cluster; if the second score value in the victoriy message is less than the second score value, a response (alive) message may be sent to the source node of the victoriy message, where the alive message for the victoriy message is a message indicating a reelect of the parent cluster master node.

For example, according to fig. 4, after the master node 21 transmits the vector message to the master node 11 and the master node 01, if the second score value in the vector message is greater than the second score value of the master node 11, the master node 11 may transmit the selection message to the master node 21; if the second score value in the vector message is greater than the second score value of master node 01, master node 01 may send an election message to master node 21.

After receiving the selection message, the master node of each sub-cluster may reply to the alive message, where the alive message for the selection message indicates that reply information of the selection message has been received. For example, referring to fig. 4, the master node 21 may transmit an alive message to the master node 11 and the master node 01, respectively, after receiving the election messages transmitted by the master node 11 and the master node 01.

After receiving the alive message for the vector message, the master node of each sub-cluster may not process the alive message and wait for receiving the messages sent by the master nodes of other sub-clusters.

After the master node of any one of the sub-clusters sends the victoriy master node to the master nodes of the other sub-clusters, if the alive message sent by the master nodes of the other sub-clusters is not received within a given time, the parent cluster is considered as the master node of the parent cluster, and at this time, the victoriy message can be sent to the master nodes of the other sub-clusters, so that the master nodes of the other sub-clusters determine the master node of the parent cluster, and the election process of the master node of the parent cluster is finished.

In some embodiments, the given time may be set according to an actual application scenario, for example, the given time may be determined according to a maximum communication delay between the respective sub-clusters.

In some embodiments of the present application, when a node of a parent cluster fails, a targeted process needs to be performed, and two cases where the parent node fails are described below respectively.

Illustratively, in the case that a slave node of a parent cluster fails, the failed slave node is deleted from the parent cluster, and member change information of the parent cluster is transmitted to the slave node of the parent cluster by using the master node of the parent cluster;

after receiving a first message sent by a master node of a first sub-cluster, a master node of the first sub-cluster is added into the parent cluster, and member change information of the parent cluster is sent to each node of the parent cluster; the first sub-cluster represents a sub-cluster to which a failed slave node belongs, and the master node of the first sub-cluster represents a master node reselected from among the nodes of the first sub-cluster.

Here, the member change information of the parent cluster may include information such as IDs of respective nodes of the parent cluster.

In some embodiments, the master node of the parent cluster may delete the failed slave node from the parent cluster, mark the ID of the corresponding child cluster as unavailable, and update the member information of the parent cluster, and then may broadcast the member change information of the parent cluster to the respective slave nodes of the parent cluster.

After the main node of the first sub-cluster fails, the main node can be reselected from the first sub-cluster, and then the main node of the first sub-cluster can send a first message to the main node of the father cluster; after the master node of the parent cluster receives the first message, the above-mentioned unavailable flag may be clear, then the member information of the parent cluster is updated, and then the member change information of the parent cluster may be broadcast to each slave node of the parent cluster.

In some embodiments, the master node of the parent cluster may also update failure information associated with the first subset group for use in electing the master node of the parent cluster.

It can be appreciated that, in the embodiment of the present application, when the slave node of the parent cluster fails, the member information of the parent cluster may be updated in time, and when the master node of the parent cluster is selected subsequently, the master node of the parent cluster may be accurately selected.

Illustratively, in the event of a failure of a master node of a parent cluster, one node is selected from the other nodes of the parent cluster as the master node of the parent cluster;

after receiving a first message sent by a master node of a second sub-cluster, adding the master node of the second sub-cluster into a father cluster, and sending member change information of the father cluster to each node of the father cluster; the second subset group represents the subset to which the failed master node belongs.

In some embodiments, in the case that the master node of the parent cluster fails, it is required to elect a node from the slave nodes of the parent cluster as the master node of the parent cluster, and the process of electing the master node of the parent cluster is described in the above description, which is not repeated herein.

Because the main node of the second sub-cluster breaks down, the main node of the second sub-cluster can be considered to be separated from the father cluster, and at this time, after the main node is selected from the second sub-cluster, the father cluster is added, and the specific flow is as follows:

when the main node of the father cluster fails, other nodes in the second sub-cluster can delete the information of the main node of the father cluster, and one node is reselected from the second sub-cluster to serve as the main node of the second sub-cluster; after the main node of the second sub-cluster is selected, the main node of the second sub-cluster can send a first message to the node of any father cluster; in some embodiments, a hash ring may be established in advance for all nodes of the parent cluster, and the master node of the second child cluster may close to select one node in a clockwise or counterclockwise direction based on the hash ring, and send the first message to the selected node.

Under the condition that the node of any one father cluster is the current master node of the father cluster, the master node of the father cluster can directly record the ID of the second sub-cluster and the address information of the current master node of the second sub-cluster. Under the condition that the node of any one father cluster is not the current master node of the father cluster, the node of any one father cluster can return the master node information of the father cluster to the current master node of the second subset cluster, and the master node of the second subset cluster can send the first message to the master node of the father cluster based on the master node information of the father cluster.

In some embodiments of the present application, writing data to each node of each sub-cluster based on the data storage architecture may include:

and sending the data writing instruction to the master node of the parent cluster, sending the data writing instruction to the master nodes of the sub-clusters by the master node of the parent cluster, and sending the data writing instruction to each slave node of the sub-cluster by the master node of each sub-cluster, so that each node of each sub-cluster writes the data to be written.

In some embodiments, after receiving the data writing instruction sent by the client, the data writing instruction indicates that data needs to be written into each node in the data storage architecture, and at this time, the data writing instruction may be sent to any node in the parent cluster; if the node receiving the data writing instruction is not the master node of the father cluster, the node receiving the data writing instruction replies the information of the master node of the father cluster to the client, and then the client can send the data writing instruction to the master node of the father cluster, and the master node of the father cluster can send the data writing instruction to the master nodes of all the sub-clusters; if the node receiving the data write instruction is the master node of the parent cluster, the master node of the parent cluster may send the data write instruction to the master nodes of the respective child clusters.

In some embodiments, after determining that all nodes in a sub-cluster successfully write data to be written, the master node of a sub-cluster returns successful write information to the master node of the parent cluster; otherwise, returning the unsuccessfully written information to the master node of the father cluster; when the master node of the father cluster receives the successful writing information sent by the master node of each sub-cluster, the successful writing information can be returned to the client; when the master node of the parent cluster receives the unsuccessful writing information sent by the master node of at least one child cluster, it may be determined that the node of the at least one child cluster fails, or it may be determined that the data is not successfully written into the at least one child cluster, at this time, a data writing instruction may be sent again to the master node of the at least one child cluster until the successful writing information sent by the master node of the at least one child cluster is received.

In some embodiments of the present application, a data reading instruction may also be obtained, where the data reading instruction carries a data reading address of any node; then, data can be read from any one of the above nodes based on the data read address.

accordingly, writing the data to be written to each node of each sub-cluster includes: establishing a hash table for each level of tag data, and constructing a multi-way tree data structure based on the hash table corresponding to each level of tag data; the at least two levels of tag data are written into each node of each sub-cluster based on the multi-drop data structure.

In the related art, the micro service technology is widely applied, and the micro service is characterized by dynamic performance, and the requirement of micro service dynamic performance cannot be met for using the traditional internet protocol (Internet Protocol, IP) to manage data (such as application instance data); thus, a data management scheme based on tag data is proposed in the embodiment of the present application instead of the IP-based data management scheme.

In some embodiments, the data to be written is application instance data, which may include the application of the cluster and the system to which the application belongs, and the application instance data may be represented as/cluster/system/app, where the symbols/represent fixed separators, cluster, system and app represent the cluster, the system, and the application, respectively, cluster represents a first level tag, system represents a second level tag, and app represents a third level tag.

Fig. 5 is a schematic diagram of a multi-tree data structure according to an embodiment of the present application, referring to fig. 5, in a cluster, hash table1-0 represents a hash address of cluster data, bucket0 and bucket1 in hash table1-0 represent different storage data, in bucket0 in hash table1-0, a key (key) is cluster0, and a value (value) is hash table1-0; in bucket1 in hashtable1-0, the key is cluster1 and the value is hashtable1-1.

Referring to fig. 5, in the system, hashtable1-0 and hashtable1-1 represent hash addresses of different system data, socket 0 and socket 1 in hashtable1-0 represent different storage data, and in socket 0 in hashtable1-0, the key is system0, and the value is hashtable2-0; in the bucket1 in the hashtable1-0, the key is system1, and the value is hashtable2-1; the bucket0 in the hash table1-1 represents storage data, and in the bucket0 in the hash table1-1, a key is system0, and a value is hash table2-2.

Referring to fig. 5, hashtable2-0, hashtable2-1, and hashtable2-2 represent hash addresses of different application data, respectively; the socket 0 in the hastable 2-0 represents storage data, and in the socket 0 in the hastable 2-0, the key is app0 and the value is ip1; the socket 0 and the socket 1 in the hashtable2-1 represent different storage data, and in the socket 0 in the hashtable2-0, the key is app1, and the value is ip2; in socket 1 in hashtable2-0, the key is app2 and the value is ip2; in socket 0 in hastable 2-2, the key is app0 and the value is ip3; here, app0, app1, and app2 are different applications, and IP1, IP2, and IP3 represent different IP addresses.

In some embodiments, in order to be compatible with the traditional IP-based data management architecture, label-to-IP conversion is required, and when network-level communication is actually performed, logic concept labels are converted into physical concept IPs, so that actual physical-level communication can be completed; for example, referring to the third level tag data of fig. 5, a correspondence relationship between tags and IPs may be established, thereby facilitating efficient storage and lookup of the correspondence relationship between tags and IPs.

In some embodiments, each tree node in the multi-tree data structure is correspondingly built with a hash table for storing corresponding primary label data.

In this embodiment of the present application, multi-level tag data may be divided according to a fixed separator, where each level of tag data corresponds to a tree node of a multi-tree data structure, and when a tree node of the multi-tree data structure is not a leaf node (a node where no child node exists), a key in the tree node is an element of a corresponding hash table, and a value is a hash address of a hash table of a next level; here, the hash address of the next-stage hash table is the hash address corresponding to the above-described next-stage tag data.

The data storage scheme based on the multi-way tree data structure has the following characteristics:

1) Tag data having the same prefix string may share the same hash table node, so that the storage space usage rate may be reduced, for example, referring to fig. 5, in the application-level tag data, bucket0 and bucket1 in hash table2-1 have the same prefix string, that is, the prefix strings of bucket0 and bucket1 in hash table2-1 are both cluster0/system1; thus, bucket0 and bucket1 in hash table2-1 share the same hash table node hash table2-1.

2) Based on the form of multi-level tag data, the number of character strings separated by adopting fixed separators is fixed, so that the height of the multi-way tree is also fixed, the searching time of each hash table is a constant level, and the complexity of the searching time based on the hash table as a whole is also a constant level.

3) When the suffix wildcard matching is needed, for example, a node which is completely matched with the prefix is needed to be found, and the pointed hash table node contains all information matched with the suffix wildcard.

4) In the related art, data storage needs to be realized by uniformly using one hash table, so that when the data volume of the hash table is large, the more keys need to be recalculated when the capacity expansion is performed, the longer the capacity expansion time is caused; when the data expansion is needed, only the hash table of the node corresponding to the new element is needed to be expanded, and the expansion time is reduced to a certain extent.

In some embodiments of the present application, at least one of the following operations may be performed on at least two levels of tag data based on a multi-way tree data structure: add, delete, modify, query.

Fig. 6 is a schematic flow chart of performing an operation of adding data to at least two levels of tag data in the embodiment of the present application, as shown in fig. 6, the flow may include:

step 601: and obtaining label data of each level.

Here, the multi-stage tag data may be divided based on the fixed separator described above to obtain the multi-stage tag data.

Step 602: and taking out the first-stage label.

For example, for data/cluster/system/app, cluster represents a first level tag, system represents a second level tag, and app represents a third level tag.

Step 603: judging whether the hash table node corresponding to the currently fetched label exists or not, if not, executing step 604; if so, step 605 is performed.

Step 604: a corresponding hash table node is created and step 605 is performed.

Here, a corresponding hash table node may be created for the currently fetched tag.

Step 605: whether the corresponding label exists in the hash table node is judged, if not, step 606 is executed, and if yes, step 607 is executed.

Step 606: the corresponding tag is inserted and then step 607 is performed.

Here, the currently fetched tag may be inserted in the hash table node, and then step 607 is performed.

Step 607: whether the tag data of each level is traversed is judged, if yes, the flow is ended, and if no, step 608 is executed.

Here, if the tag data of each level has been fetched, this indicates that the tag data of each level has been traversed, otherwise, it indicates that the tag of each level has not been traversed.

Step 608: the next level of tag data is fetched, and then, returns to step 603.

Fig. 7 is a schematic flow chart of performing a query data operation on at least two levels of tag data in an embodiment of the present application, as shown in fig. 7, the flow may include:

step 701: and obtaining label data of each level.

Step 702: and taking out the first-stage label.

Step 703: whether the hash table node corresponding to the currently fetched tag exists or not is judged, if yes, step 704 is executed, and if not, step 708 is executed.

Step 704: whether the corresponding label exists in the hash table node is judged, if yes, step 705 is executed, and if no, step 708 is executed.

Step 705: whether the tag data of each level is traversed is judged, if yes, step 706 is executed, and if no, step 707 is executed.

Step 706: and determining that the corresponding label is queried, and ending the flow.

Here, after determining that the corresponding tag is queried, the corresponding tag data may be read out; in practical application, corresponding data can be read from the hash table node where the tag data is located.

Step 707: the next level tag is fetched and then returns to step 703.

Step 708: and determining that the corresponding label is not queried, and ending the flow.

In some embodiments, when the data deleting operation is performed on the at least two levels of tag data, the corresponding data may be queried first, and then the corresponding data may be deleted in the corresponding hash node.

In some embodiments, when the data modification operation is performed on the at least two levels of tag data, the corresponding data may be queried first, and then the corresponding data may be modified in the corresponding hash node.

On the basis of the data management method provided in the foregoing embodiment, the embodiment of the present application further provides a data management device; fig. 8 is a schematic diagram of an optional component structure of a data management device according to an embodiment of the present application, and as shown in fig. 8, the data management device 800 may include:

A building module 801, configured to build a data storage architecture, where the data storage architecture includes a parent cluster and a plurality of sub-clusters, each sub-cluster in the plurality of sub-clusters includes a master node and at least one slave node, a node of the parent cluster is a master node of the plurality of sub-clusters, and a node of the parent cluster includes a master node of the parent cluster and at least one slave node of the parent cluster;

a processing module 802 for writing data to each node of each sub-cluster based on the data storage architecture.

In some embodiments of the present application, the establishing module 801 is configured to establish a data storage architecture, including:

determining main nodes and health degree indexes of all sub-clusters;

In some embodiments of the present application, the establishing module 801 is further configured to derive the health indicator of the master node of each sub-cluster according to at least one of the following: the number of faults, the duration of at least one fault, the duration of the last fault from the current time.

In some embodiments of the present application, the establishing module 801 is configured to select, from the master nodes of the respective sub-clusters, a master node as the master node of the parent cluster based on the health indicator of the respective sub-clusters, including:

In some embodiments of the present application, the establishing module 801 is configured to select, from the master nodes of the respective sub-clusters, a master node as the master node of the parent cluster based on the health indicator and the first score value of the respective sub-clusters, including:

In some embodiments of the present application, the establishing module 801 is configured to, based on data interaction of the master node of each sub-cluster, cause the master node of each sub-cluster to obtain a first message of each sub-cluster, including:

selecting seed clusters from the sub-clusters;

In some embodiments of the present application, the establishing module 801 is further configured to:

In some embodiments of the present application, the processing module 802 is further configured to:

and reading data from any one node based on the data reading address.

In some embodiments of the present application, the processing module 802, configured to write data to each node of each sub-cluster based on the data storage architecture, includes:

writing the data to be written to each node of each sub-cluster comprises:

In practical applications, both the establishing module 801 and the processing module 802 may be implemented by using a processor, where the processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, a controller, a microcontroller, and a microprocessor. It will be appreciated that the electronic device implementing the above-described processor function may be other, and embodiments of the present application are not limited.

It should be noted that the description of the above device embodiments is similar to the description of the method embodiments described above, with similar advantageous effects as the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be noted that, in the embodiment of the present application, if the above-mentioned data management method is implemented in the form of a software functional module, and sold or used as a separate product, the data management method may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or portions contributing to the prior art may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a terminal, a server, etc.) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, the embodiment of the application further provides a computer program product, which comprises computer executable instructions for implementing any of the data management methods provided in the embodiment of the application.

Accordingly, embodiments of the present application further provide a computer storage medium having stored thereon computer-executable instructions for implementing any of the data management methods provided in the foregoing embodiments.

An electronic device is further provided in the embodiments of the present application, and fig. 9 is a schematic diagram of an optional composition structure of the electronic device provided in the embodiments of the present application, as shown in fig. 9, where the electronic device 900 includes:

a memory 901 for storing executable instructions;

and a processor 902, configured to implement any one of the data management methods when executing the executable instructions stored in the memory 901.

The processor 902 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, a controller, a microcontroller, and a microprocessor.

The computer readable storage medium/Memory may be a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable programmable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable programmable Read Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic random access Memory (Ferromagnetic Random Access Memory, FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Read Only optical disk (Compact Disc Read-Only Memory, CD-ROM); but may also be various terminals such as mobile phones, computers, tablet devices, personal digital assistants, etc., that include one or any combination of the above-mentioned memories.

It should be noted here that: the description of the storage medium and apparatus embodiments above is similar to that of the method embodiments described above, with similar benefits as the method embodiments. For technical details not disclosed in the embodiments of the storage medium and the apparatus of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be appreciated that reference throughout this specification to "some embodiments" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrase "in some embodiments" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application. The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units; can be located in one place or distributed to a plurality of network units; some or all of the units may be selected according to actual needs to achieve the purposes of the embodiments of the present application.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated in one unit; the integrated units may be implemented in hardware or in hardware plus software functional units.

Alternatively, the integrated units described above may be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partly contributing to the related art, embodied in the form of a software product stored in a storage medium, including several instructions for causing an apparatus automatic test line to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The methods disclosed in the several method embodiments provided in the present application may be arbitrarily combined without collision to obtain a new method embodiment.

The features disclosed in the several method or apparatus embodiments provided in the present application may be arbitrarily combined without conflict to obtain new method embodiments or apparatus embodiments.

The foregoing is merely an embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data management, the method comprising:

determining main nodes and health degree indexes of all sub-clusters; the health degree index is used for reflecting the health degree of the main nodes of each sub-cluster;

based on the health degree index of each sub-cluster, selecting a master node from the master nodes of each sub-cluster as the master node of the father cluster;

establishing a data storage architecture based on the master nodes of the sub-clusters and the master nodes of the father clusters, wherein the data storage architecture comprises the father clusters and a plurality of sub-clusters, each sub-cluster comprises one master node and at least one slave node, the nodes of the father clusters are the master nodes of the plurality of sub-clusters, and the nodes of the father clusters comprise one master node of the father clusters and at least one slave node of the father clusters;

writing data to each node of each sub-cluster based on the data storage architecture;

the selecting a master node as the master node of the parent cluster from the master nodes of the respective sub-clusters based on the health degree index of the respective sub-clusters includes:

Based on the data interaction of the main nodes of the sub-clusters, the main node of each sub-cluster is enabled to acquire a second grading value of the main node of each sub-cluster, wherein the second grading value is the sum of the health degree index and a first grading value, and the first grading value is a preconfigured value;

and based on the second grading values of the main nodes of the sub-clusters, the main nodes of the parent clusters are elected by the main nodes of the sub-clusters.

2. The method according to claim 1, wherein the method further comprises:

3. The method according to claim 1, wherein selecting one of the master nodes of the respective sub-clusters as the master node of the parent cluster based on the health index of the respective sub-clusters comprises:

4. A method according to claim 3, wherein the causing the master node of each sub-cluster to obtain the first message of each sub-cluster based on the data interaction of the master nodes of each sub-cluster comprises:

selecting seed clusters from the sub-clusters;

5. The method according to claim 1, wherein the method further comprises:

after receiving a first message sent by a master node of a first sub-cluster, a master node of the first sub-cluster is added into the parent cluster, and member change information of the parent cluster is sent to each node of the parent cluster; the first subset group represents a subset group to which the failed slave node belongs, the master node of the first subset group represents a master node reselected from each node of the first subset group, and the first message represents a message of joining a parent group.

6. The method according to claim 1, wherein the method further comprises:

7. The method according to claim 1, wherein the method further comprises:

and reading data from any one node based on the data reading address.

8. The method of any of claims 1-7, wherein the writing data to each node of each sub-cluster based on the data storage architecture comprises:

9. The method of claim 8, wherein the data to be written comprises at least two levels of tag data;

writing the data to be written to each node of each sub-cluster comprises:

10. The method of claim 9, wherein the hash table of any one level of tag data includes a hash address corresponding to the next level of tag data in the case where the any one level of tag data is not the lowest level of tag data.

11. The method according to claim 9, wherein the method further comprises:

12. A data management apparatus, the apparatus comprising:

the building module is used for determining the main node and the health degree index of each sub-cluster; the health degree index is used for reflecting the health degree of the main nodes of each sub-cluster; based on the health degree index of each sub-cluster, selecting a master node from the master nodes of each sub-cluster as the master node of the father cluster; establishing a data storage architecture based on the master nodes of the sub-clusters and the master nodes of the parent clusters, wherein the data storage architecture comprises a parent cluster and a plurality of sub-clusters, each sub-cluster comprises a master node and at least one slave node, the nodes of the parent cluster are the master nodes of the sub-clusters, and the nodes of the parent cluster comprise a master node of the parent cluster and at least one slave node of the parent cluster;

The processing module is used for writing data into each node of each sub-cluster based on the data storage architecture;

the establishing module is configured to select, based on the health degree index of each of the sub-clusters, a master node from among the master nodes of each of the sub-clusters as a master node of the parent cluster, where the establishing module includes:

13. An electronic device, the electronic device comprising:

a memory for storing executable instructions;

a processor for implementing the data management method of any one of claims 1 to 11 when executing executable instructions stored in said memory.

14. A computer readable storage medium storing executable instructions for implementing the data management method of any one of claims 1 to 11 when executed by a processor.