CN107277111B - A kind of company-data processing method, master node, slave node and cluster - Google Patents
A kind of company-data processing method, master node, slave node and cluster Download PDFInfo
- Publication number
- CN107277111B CN107277111B CN201710357042.4A CN201710357042A CN107277111B CN 107277111 B CN107277111 B CN 107277111B CN 201710357042 A CN201710357042 A CN 201710357042A CN 107277111 B CN107277111 B CN 107277111B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- information packet
- synchronizing information
- host
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1034—Reaction to server failures by a load balancer
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The present invention provides a kind of company-data processing method, master node, slave node and clusters, by a kind of company-data processing method of offer the data of all nodes of cluster are constantly updated in a circulating manner, host node and other snapshots from node each can be obtained from node, and inquiry service independently can be externally provided, and then by the calculating of cluster and inquire all nodes in pressure dissipation to cluster；Host node and other snapshots from node are remained with from node due to each, even if delay machine occur in host node or other nodes, each node remains able to externally provide complete, correct data, to significantly increase the robustness of entire cluster.
Description
Technical field
The present invention relates to Distributed Calculation field more particularly to a kind of company-data processing method, master node, slave node and
Cluster.
Background technology
The method of synchronization of data in currently used cluster will be as shown in Figure 1, other nodes typically in cluster incite somebody to action this
The data on ground are sent to host node, and storage and joint account, and unified externally offer query function are carried out by host node, this existing
There is scheme stored at least in following three problems：
(1) excessive or when data volume is excessive from node, host node processing pressure can be bigger；
(2) there was only host node just has the partial data of entire cluster, could externally provide the inquiry of cluster operating status, because
It is also bigger that this host node inquires pressure；
(3) once there is delay machine in host node, and new host node can lose the data of original host node, it is imperfect data occur
The case where.
Invention content
In order to solve the above-mentioned technical problem, the present invention propose a kind of company-data processing method, master node, slave node and
Cluster.The present invention is specifically to be realized with following technical solution：
In a first aspect, a kind of company-data processing method, the data processing method include：
Host node generates synchronizing information packet, and the synchronizing information packet is distributed in node cycle so that the synchronous letter
Breath packet reaches next node along the node cycle；The node cycle is made of a host node and no less than one from node,
The node cycle includes whole nodes in the cluster；
The synchronizing information packet carries out first round transmission in the node cycle：After often reaching a node, present node
The data of own local are changed according to the first preset rules, and synchronizing information packet is carried out more according to the second preset rules
Newly, the updated synchronizing information packet is distributed to again on node cycle and continues to transmit so that the synchronizing information packet is along institute
It states node cycle and reaches next node；
The synchronizing information packet is obtained as starting point using the host node, executes the synchronizing information packet in the node cycle
Second wheel transmit：The present node for obtaining synchronizing information packet changes the data of own local according to third preset rules
And it is updated the data to the publication of the communication node of the present node；According to the 4th preset rules in the synchronizing information packet
Content is deleted, and the synchronizing information packet after deleting is distributed on node cycle and continues to transmit so that the synchronous letter
Breath packet reaches next node along the node cycle；
After the host node obtains the synchronizing information packet again, the synchronizing information packet is abandoned, returns and executes institute
State the step of host node generates synchronizing information packet.
Second aspect, a kind of host node, the host node include：
Synchronizing information packet generation module, for generating synchronizing information packet；
Synchronizing information packet transmit control module, for by synchronizing information packet publisher node ring so that the synchronizing information
Packet reaches next node along the node cycle；The node cycle is made of a host node and no less than one from node, institute
It includes whole nodes in cluster to state node cycle；
Data processing module, for being changed to the data of own local according to third preset rules and to the master
The communication node publication of node updates the data；
Synchronizing information packet changes module, for being deleted to the content in the synchronizing information packet according to the 4th preset rules
Subtract；
Synchronizing information packet discard module, for abandoning the synchronizing information after the second wheel transmits in synchronizing information packet
Packet.
The third aspect, from node, the node includes one kind：
Synchronizing information packet acquisition module, for obtaining synchronizing information packet；
Synchronizing information packet transmits control module, for synchronizing information packet being distributed in node cycle so that the synchronous letter
Breath packet reaches next node along the node cycle；The node cycle is made of a host node and no less than one from node,
The node cycle includes whole nodes in cluster；
Local data update module, for being changed to the data of own local according to the first preset rules；
Synchronizing information packet update module, for being updated to synchronizing information packet according to the second preset rules；
Data processing module, for the data of own local are changed according to third preset rules and to it is described from
The communication node publication of node updates the data；
Synchronizing information packet changes module, for being deleted to the content in the synchronizing information packet according to the 4th preset rules
Subtract.
Fourth aspect, a kind of cluster, the cluster include above-mentioned host node and above-mentioned from node.
The present invention provides a kind of company-data processing method, master node, slave node and clusters, have following beneficial to effect
Fruit：
The present invention is by providing a kind of company-data processing method so that the data of all nodes of cluster can be with cycle
Mode is constantly updated, and host node and other snapshots from node each can be obtained from node, and can be independent right
Outer offer inquiry service, and then by the calculating of cluster and inquire all nodes in pressure dissipation to cluster；Due to each from node
Host node and other snapshots from node are remained with, therefore, even if delay machine occur in host node or other nodes, each node is still
Complete, correct data so can be externally provided, to significantly increase the robustness of entire cluster.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the method for synchronization schematic diagram for the data that background of invention provides；
Fig. 2 is a kind of applied environment figure of company-data processing method provided in an embodiment of the present invention；
Fig. 3 is a kind of company-data process flow figure provided in an embodiment of the present invention；
Fig. 4 is the method flow diagram that host node provided in an embodiment of the present invention generates synchronizing information packet；
Fig. 5 is the method provided in an embodiment of the present invention changed to the data of own local according to the first preset rules
Flow chart；
Fig. 6 is the method flow provided in an embodiment of the present invention being updated to synchronizing information packet according to the second preset rules
Figure；
Fig. 7 be it is provided in an embodiment of the present invention the data of own local are changed according to third preset rules and to
The communication node of present node issues the method flow diagram updated the data；
Fig. 8 is provided in an embodiment of the present invention to be deleted to the content in the synchronizing information packet according to the 4th preset rules
The method flow diagram subtracted；
Fig. 9 is the complete data handling procedure schematic diagram of cluster provided in an embodiment of the present invention；
Figure 10 is the method flow diagram of data query provided in an embodiment of the present invention；
Figure 11 is host node block diagram provided in an embodiment of the present invention；
Figure 12 is synchronizing information packet generation module frame provided in an embodiment of the present invention；
Figure 13 is data processing module block diagram provided in an embodiment of the present invention；
Figure 14 is synchronizing information packet change module frame chart provided in an embodiment of the present invention；
Figure 15 is provided in an embodiment of the present invention from node block diagram；
Figure 16 is local data update module block diagram provided in an embodiment of the present invention；
Figure 17 is synchronizing information packet update module block diagram provided in an embodiment of the present invention；
Figure 18 is server architecture schematic diagram provided in an embodiment of the present invention.
Specific implementation mode
In order to enable those skilled in the art to better understand the solution of the present invention, below in conjunction in the embodiment of the present invention
Attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only
The embodiment of a part of the invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people
The every other embodiment that member is obtained without making creative work should all belong to the model that the present invention protects
It encloses.
It should be noted that term " first " in description and claims of this specification and above-mentioned attached drawing, "
Two " etc. be for distinguishing similar object, without being used to describe specific sequence or precedence.It should be appreciated that using in this way
Data can be interchanged in the appropriate case, so as to the embodiment of the present invention described herein can in addition to illustrating herein or
Sequence other than those of description is implemented.In addition, term " comprising " and " having " and their any deformation, it is intended that cover
It includes to be not necessarily limited to for example, containing the process of series of steps or unit, method, system, product or equipment to cover non-exclusive
Those of clearly list step or unit, but may include not listing clearly or for these processes, method, product
Or the other steps or unit that equipment is intrinsic.
An embodiment of the present invention provides a kind of applied environment figures of company-data processing method, as shown in Fig. 2, the cluster
Data processing method can be applied among the distributed system being made of terminal 110 and server cluster 120.Server cluster
120 include a host node and at least one from node.Terminal 110 can have multiple, such as terminal 110 (1) and terminal 110
(2), each terminal constitutes a client node, and each client node can be with any one in the server cluster 120
Node communicates, to the communication node as some node.
Under above application environment, the embodiment of the present invention provides a kind of company-data processing method, the method such as Fig. 3 institutes
Show, specifically includes：
S101. host node generates synchronizing information packet, and the synchronizing information packet is distributed in node cycle so that described
Synchronizing information packet reaches next node along the node cycle.
Specifically, the node cycle is made of a host node and no less than one from node, and the node cycle includes institute
State whole nodes in cluster.
S102. the synchronizing information packet carries out first round transmission in the node cycle.
Specifically, after synchronizing information packet often reaches a node, obtain the node of synchronizing information packet in order to currently save
Point.The present node changes the data of own local according to the first preset rules, and according to the second preset rules pair
Synchronizing information packet is updated, and the updated synchronizing information packet is distributed to again on node cycle and continues to transmit so that institute
It states synchronizing information packet and reaches next node along the node cycle.
During synchronizing information packet carries out first round transmission in the node cycle, the content of synchronizing information packet is continuous
It is updated abundant, and each node is also according to the data of the abundant in content own local of synchronizing information packet.It is transmitted when the first round
After, content in synchronizing information packet is i.e. by all from node updates one time on node cycle, to include entire cluster
Whole valid data.
S103. the synchronizing information packet is obtained as starting point using the host node, executes the synchronizing information packet in the section
The second wheel in point ring transmits.
Second wheel is transmitted to be started with host node, and is terminated after node after all on node cycle, no matter main section
For point still from node, the node for getting synchronizing information packet is present node.Host node and it is performed both by operations described below from node：
Present node changes the data of own local according to third preset rules and to the present node
Communication node publication updates the data；The content in the synchronizing information packet is deleted according to the 4th preset rules, and will be deleted
The synchronizing information packet after subtracting, which is distributed on node cycle, to be continued to transmit so that the synchronizing information packet is arrived along the node cycle
Up to next node.No matter host node or from node, the client node being in communication with is its communication node.
Synchronizing information packet carried out in the node cycle second wheel transmit during, host node and from node all in accordance with
Obtained synchronizing information packet further enriches its own local data and is updated the data to the publication of its communication node.By to logical
Letter node publication, which updates the data, can make communication node timely receive the data of newest cluster, with the same stepping of company-data
Row update.Node can be kept in touch between communicating with node by heartbeat.Specifically, from communication node to the node period
Property heartbeat is actively sent to maintain to write to each other.
During the second wheel transmits, often pass through a node, the data in synchronizing information packet are deleted, to subtract
Lack the data volume of synchronizing information packet, reduces network burden.
S104. after the host node obtains the synchronizing information packet again, the synchronizing information packet is abandoned, return is held
The step of row host node generates synchronizing information packet (S101).
After the second wheel transmits, the synchronizing information packet comes back to host node, each in being transmitted by the second wheel
Node deletes data in synchronizing information packet, come back in the synchronizing information packet of host node no longer existed it is valuable
Data, therefore abandoned by host node.Host node repeats S101 again, makes entire cluster by repeating S101
Data in the communication node of node in data and cluster can be continuously updated.
An embodiment of the present invention provides a kind of company-data processing methods, by host node cycle generate synchronizing information
Packet, and synchronizing information packet is made to carry out two wheels on node cycle and transmit to realize the timely benefit of each node own local data
Fill, while the data for also achieving communication node timely update so that the data of entire cluster and in cluster
Data in the communication node of node can be continuously updated.Each node in cluster can preserve it as a result,
The snapshot of the node data of its node, to have the ability independently to be provided out service, to mitigate the pressure of host node.Further
Ground avoids since each node can preserve the snapshot of the node data of other nodes and there was only main section in the prior art
The case where point storage total data, to once occur host node or some from the delay machine of node, cluster can also be adjusted in time
It is whole, loss of data is avoided the occurrence of, to improve the robustness of cluster.
Another embodiment of the present invention provides a kind of company-data processing method, including：
S201. host node generates synchronizing information packet, and the synchronizing information packet is distributed in node cycle so that described
Synchronizing information packet reaches next node along the node cycle.
Specifically, the node cycle is made of a host node and no less than one from node, and the node cycle includes institute
State whole nodes in cluster.
Referring to FIG. 4, it illustrates the method flow diagrams that host node generates synchronizing information packet, including：
S2011. the node data of host node is obtained.
S2012. the communication data of host node is obtained, the communication data of the host node is from the communication node of host node
The communication node of the data received, the host node is the client node communicated with host node.
S2013. synchronizing information packet is generated according to the node data of the host node and communication data.
S202. the synchronizing information packet carries out first round transmission in the node cycle.
Specifically, present node changes the data of own local according to the first preset rules, and pre- according to second
If rule is updated synchronizing information packet, the updated synchronizing information packet is distributed to again on node cycle and continues to transmit
So that the synchronizing information packet reaches next node along the node cycle.
S203. the synchronizing information packet is obtained as starting point using the host node, executes the synchronizing information packet in the section
The second wheel in point ring transmits.
Present node changes the data of own local according to third preset rules and to the present node
Communication node publication updates the data；The content in the synchronizing information packet is deleted according to the 4th preset rules, and will be deleted
The synchronizing information packet after subtracting, which is distributed on node cycle, to be continued to transmit so that the synchronizing information packet is arrived along the node cycle
Up to next node.
It is executed according to the first preset rules, the second preset rules, third preset rules and the 4th preset rules following
In the explanation of relevant operation, it is illustrated with following situations：Node in node cycle is sequentially numbered, and node cycle includes
Node 1-7, present node are node 5, and the communication node of present node is client node 5-6, and whole nodes is logical in node cycle
Letter node shares 9 (client node 1-9).
Specifically, referring to FIG. 5, it illustrates the sides changed to the data of own local according to the first preset rules
Method flow chart, including：
S2021. the data in the synchronizing information packet are extracted.
The data that data in the synchronizing information packet that present node 5 receives are node 1-4 and client node 1-4.
S2022. the snapshot of the node data of other nodes is generated according to the data in the synchronizing information packet.
Present node is node 5, the data in node 1-4 and the data in client node 1-4 be stored in it is other
Data at node, therefore, generation include the snapshot of the data in node 1-4 and the data in client node 1-4.
S2023. the snapshot is stored in own local.
After the snapshot to be stored in the own local of present node (node 5), present node own local includes to work as
The node data (node 5) of front nodal point, the communication data (client node 5-6) of present node and snapshot are (including in node 1-4
Data and client node 1-4 in data).
Specifically, referring to FIG. 6, it illustrates the method streams being updated to synchronizing information packet according to the second preset rules
Cheng Tu, including：
S20210. the node data of present node is obtained.
S20220. the communication data of present node is obtained, the communication data of the present node is from the logical of present node
The data received at letter node, the communication node of the present node is the client node communicated with present node.
By taking the node of present node is node 5 as an example, the node data of present node is the data of node 5, works as prosthomere
The communication data of point is the data received from client node 5-6.
S20230. according to the communication data of the node data of present node and the present node to the synchronizing information packet
It is updated.
It is added in synchronizing information packet by the data of node 5 and from the data that client node 5-6 is received.
Specifically, referring to FIG. 7, it illustrates being changed to the data of own local according to third preset rules and
The method flow diagram updated the data is issued to the communication node of present node, including：
S2031. the data in synchronizing information packet are obtained.
S2032. the snapshot of the node data of other nodes is generated according to the data in synchronizing information packet.
Include node 5-7 and client node for present node (node 5), in the synchronizing information packet obtained in S2031
When the data of 5-9, wherein client node 5-6 is the data of the communication data of present node, node 6-7 and client node 7-9
To be stored in the node data of other nodes, therefore, snapshot is generated according to node 6-7 and client node 7-9.
S2033. the snapshot stored according to the snapshot update own local.
After step S2023,5 own local of node is stored with node data (node 5), the present node of present node
Communication data (client node 5-6) and snapshot (including the data in node 1-4 and data in client node 1-4),
The update of snapshot is carried out on this basis, then the snapshot stored increases the data of node 6-7 and client node 7-9 again.As it can be seen that
In the data for whole client nodes that 5 own local of node stores node cycle whole node and is communicated with node cycle.
S2034. it obtains the node data of present node and is stored in the node of other nodes of present node own local
The snapshot of data.
S2035. it is updated the data according to the content generation got.
S2036. described update the data is distributed to the communication node of present node.
It is described update the data may include node own local storage total data include node by taking node 5 as an example
The data of 1-7 and the data of client node 1-9, identical update the data is issued to client node 5 and 6.Another kind can
In capable embodiment, update the data can also be exclude from will with the data of the client node of the communication of present node it
Outer data, by taking node 5 as an example, if necessary to client node 5 publication update the data, then update the data including node 1-7,
The data of client node 1-4 and client node 6-9；It is updated the data if necessary to be issued to client node 6, then updated data package
The data of node 1-7, client node 1-5 and client node 7-9 are included.
Further, referring to FIG. 8, it illustrates according to the 4th preset rules to the content in the synchronizing information packet into
The method flow diagram that row is deleted, including：
S20310. the node data of present node in the synchronizing information packet is deleted.
S20320. the communication data of present node in the synchronizing information packet is deleted.
After node 5 obtains synchronizing information packet, deleted in synchronizing information packet after executing step S2031-S2036
Node 5 data and client node 5-6 data.
It is worth noting that, carrying out often reaching a section during the second wheel transmits on node cycle in synchronizing information packet
Point can be deleted the communication data of the node data of present node by present node, therefore obtained in step S2032 interior joints 5
There was only the data of node 5-7 and client node 5-9 in synchronizing information packet, and node 1-4 and client node 1-4 are in synchronization
Packet is deleted when reaching node 1-4.
S204. after the host node obtains the synchronizing information packet again, the synchronizing information packet is abandoned, return is held
The step of row host node generates synchronizing information packet.
Data processing method in the embodiment of the present invention can handle various data, be more clear by taking statistical data as an example
Content in the clear elaboration embodiment of the present invention describes its complete Random data processing flow by taking following clusters as an example.
There are three server nodes for the cluster：Server_A, server_B, server_C, based on wherein server_A
Node, server_B and server_C are from node, and there are two client nodes for the cluster tool：client_A,client_B.
The machine metric is sent to server_A by client_A, and the machine metric is sent to server_B by client_B；Node cycle
Order be server_A->server_B->Server_C, metric indicates statistical data in this example.
Fig. 9 is referred to, it illustrates the complete data handling procedures of above-mentioned cluster, now general to above-mentioned data handling procedure
It states as follows：
1. local statistical data (metric) by heart beat cycle is sent to by all client nodes in cluster
Its corresponding node.As can be seen from Fig. 9, client_A is sent out local metric by heartbeat (HeartBeat (client_A))
It send to server_A with the communication node as server_A, client_B will by heartbeat (HeartBeat (client_B))
Local metric is sent to the communication node that server_B has become server_B.
2. host node server_A is periodically by the statistical data of own local, the statistics with the client_A received
Data are packaged together generates synchronizing information packet (MetricSyncMsg), is sent on node cycle.As can be seen from Fig. 9, it is initially generated
MetricSyncMsg in metric include server_A metric and client_A metric.
3.MetricSyncMsg is when the upload of the first torus is passed, after the node on ring receives, according to MetricSyncMsg
Data update own local statistical data, while by own local statistical data and the statistical data of the communication node received
It is added to MetricSyncMsg, is further continued for being put on ring, next node is allowed to handle.
As can be seen from Fig. 9, the metric in the MetricSyncMsg that server_B is received includes the metric of server_A
And the metric of client_A, therefore the statistical data of own local is updated to both include server_B's by server_B
Metric, and include the metric of its communication node client_B, further comprise server_A's and client_A
The snapshot of metric；And by the statistical data of the metric of own local statistical data server_B and its communication node
The metric of client_B is added MetricSyncMsg and is therefore transferred in the MetricSyncMsg of server_C and contains
The metric of server_A, client_A, server_B and client_B.
As can be seen from Fig. 9, after the MetricSyncMsg that server_C is received, the statistical data of own local is updated to
Not only included the metric of server_C, but also the metric including server_A, client_A, server_B and client_B
Snapshot；And MetricSyncMsg is added in the metric of own local statistical data server_C, therefore,
The metric of server_A, client_A, server_B, client_B and server_C are contained in MetricSyncMsg.
4. terminating when the first round transmits, when synchronizing information packet returns to host node server_A, that is, the biography of the second torus is opened
It passs.The synchronizing information packet that host node server_A is received contains the statistical data of all nodes and its communication node in cluster
(metric of server_A, client_A, server_B, client_B and server_C, the node on ring continue according to same
The statistical data of data update own local in step packet with obtain complete statistical data (all nodes in cluster and
The statistical data of its communication node), and updated the data to the publication of its communication node, and its own system is deleted from synchronizing information packet
It counts and its statistical data of communication node, then synchronizing information packet is continued to be put on ring.
As can be seen from Fig. 9, contained in the MetricSyncMsg that server_A is obtained server_A, client_A,
The metric of server_B, client_B and server_C, therefore, the statistical data of own local is updated to both include
The metric of server_A, and include the metric of its communication node client_A, further include server_B, client_B and
The metric of server_C.So far, server_A has obtained complete statistical data, and can the statistical data be sent to it
Communication node client_A.Further, to reduce traffic volume, the metric of client_A itself can not also be sent.Hereafter,
Delete the metric of server_A and its communication node client_A in MetricSyncMsg.
As can be seen from Fig. 9, contained in the MetricSyncMsg that server_B is obtained server_B, client_B and
The metric of server_C, in the first round transmits, server_B has included server_B, client_B, server_A
And the metric of client_A；The metric of server_C is this time obtained in MetricSyncMsg again, so far server_B
Complete statistical data has been obtained, and the statistical data can be sent to its communication node client_B；Further, to subtract
Few traffic volume, can not also send the metric of client_B itself.Hereafter, delete MetricSyncMsg in server_B and
The metric of its communication node client_B.
As can be seen from Fig. 9, the metric that server_C is contained in the MetricSyncMsg that server_C is obtained,
During one wheel transmits, the statistical data of server_C has included server_A, client_A, server_B and client_B
Metric；So far server_C has obtained complete statistical data, since it does not have communication node, can not send
Statistical data.Hereafter, the metric of sserver_C in MetricSyncMsg is deleted, so far, MetricSyncMsg no longer includes
Any metric.
5. host node server_A receives MetricSyncMsg for the second time, abandon, no longer transmitted on ring, epicycle synchronizes
It finishes.All there are one complete statistical data for all nodes in cluster at this time.
6. next cycle, host node server_A is waited for continue to generate and send MetricSyncMsg.
An embodiment of the present invention provides a kind of company-data processing method and specifically illustrate by synchronizing information packet two
During how secondary transmission enables each node to obtain the data of entire cluster and client's point for being communicated with the node in cluster
Data, and how the process updated the data in time to client node, to be further elaborated with why in cluster
Each node can preserve the snapshot of the node data of other nodes, absolutely prove each node in the embodiment of the present invention
Have the ability independently to be provided out service to mitigate the pressure of host node.
Further, the embodiment of the present invention additionally provides a kind of company-data processing method, the data processing method ginseng
Examine the above embodiments make node all in cluster be owned by complete data (data of entire cluster and in cluster
Node communication client node in data), on this basis, can also include the following steps：
Further include deleting before the present node changes the data of own local according to the first preset rules
The step of snapshot of the node data of other nodes of present node own local storage.Executing step S104 or step
After rapid S204, if obtaining synchronizing information packet again from node, the number of nodes of other nodes of own local storage is first deleted
According to snapshot, only preserve its own node data and its its communication node that can be obtained in real time data.
If host node is deleted in cluster in the cluster, according to preset node selection algorithm, in the weight from node
New selection host node.If added in cluster new slave node or delete it is old from node, by the dynamic for causing node cycle update with
So that node cycle includes whole nodes in cluster always.The dynamic change of node cycle does not influence at the data of the embodiment of the present invention
Reason method, if that is, synchronizing information packet can completely be carried out in node cycle two-wheeled transmission may make section all in cluster
Point is owned by complete data, and can be updated the data in time to communication node publication.
In the cluster, since each node is owned by complete data, can be with the number of independent process user
According to inquiry request, referring to FIG. 10, it illustrates the method flow diagrams that any one node carries out data query, including：
S301. node obtains the data inquiry request of client node.
S302. the snapshot of the node data of the other nodes preserved according to the node data of the node and the node is given birth to
It is responded at data query.
S303. data query response is back to the client node.
An embodiment of the present invention provides a kind of company-data processing methods, and node is adapted to by the dynamic change of node cycle
Transformation, to ensure being normally carried out for data handling procedure, do not influenced by node transformation, once occur host node or certain
A delay machine from node, cluster can also adjust in time, avoid the occurrence of loss of data, to improve the robustness of cluster.Often
One node independently can externally provide inquiry service, reduce the dependence for host node.
The present invention also provides a kind of host nodes, and as described in Figure 11, the host node includes：
Synchronizing information packet generation module 401, for generating synchronizing information packet.It can be used for executing step S101 and S201.
Synchronizing information packet transmit control module 402, for by synchronizing information packet publisher node ring so that the synchronization
Packet reaches next node along the node cycle；The node cycle is by a host node and no less than one from node structure
At the node cycle includes whole nodes in cluster.It can be used for participating in executing step S101, S103, S201 and S203.
Data processing module 403, for being changed to the data of own local according to third preset rules and to institute
The communication node publication for stating host node updates the data.It can be used for executing step S103 and S203.
Synchronizing information packet change module 404, for according to the 4th preset rules to the content in the synchronizing information packet into
Row is deleted.It can be used for executing step S103 and S203.
Synchronizing information packet discard module 405, for abandoning the synchronous letter after the second wheel transmits in synchronizing information packet
Breath packet.It can be used for executing step S104 and S204.
Specifically, 2 are please referred to Fig.1, the synchronizing information packet generation module 401 includes：
First data capture unit 4011, the node data for obtaining host node.It can be used for executing step S2011.
Second data capture unit 4012, the communication data of the communication data for obtaining host node, the host node are
The data received from the communication node of host node, the communication node of the host node are the client's section communicated with host node
Point.It can be used for executing step S2012.
Synchronizing information packet generation unit 4013, for synchronous with communication data generation according to the node data of the host node
Packet.It can be used for executing step S2013.
Specifically, 3 are please referred to Fig.1, the data processing module 403 includes：
Data capture unit 4031, for obtaining the data in the synchronizing information packet.It can be used for executing step S2031.
Snapshot generation unit 4032, for generating other number of nodes from node according to the data in the synchronizing information packet
According to snapshot.It can be used for executing step S2032.
Updating unit 4033, the snapshot for being stored according to the snapshot update own local.It can be used for executing step
S2033。
Local data acquiring unit 4034, for obtain host node node data and be stored in host node own local
The snapshot of the node data of other nodes.It can be used for executing step S2034.
Generation unit 4035 is updated the data, for being updated the data according to the content generation got.It can be used for executing step
S2035。
Release unit 4036, for described update the data to be distributed to the communication node of host node.It can be used for executing step
S2036。
Specifically, 4 are please referred to Fig.1, the synchronizing information packet change module 404 includes：
First deleting unit 4041, the node data for deleting host node in the synchronizing information packet.It can be used for executing
Step S20310.
Second deleting unit 4042, the communication data for deleting host node in the synchronizing information packet.It can be used for executing
Step S20320.
The present invention also provides one kind from node, as shown in figure 15, described to include from node：
Synchronizing information packet acquisition module 501, for obtaining synchronizing information packet.It can be used for executing step S102 and S202.
Synchronizing information packet transmits control module 502, for synchronizing information packet being distributed in node cycle so that described same
It walks packet and reaches next node along the node cycle；The node cycle is by a host node and no less than one from node structure
At the node cycle includes whole nodes in cluster.It can be used for executing step S102, S103, S202 and S203.
Local data update module 503, for being changed to the data of own local according to the first preset rules.It can use
In execution step S102 and S202.
Synchronizing information packet update module 504, for being updated to synchronizing information packet according to the second preset rules.It can be used for
Execute step S102 and S202.
Data processing module 505, for being changed to the data of own local according to third preset rules and to institute
It states and is updated the data from the publication of the communication node of node.It can be used for executing step S103 and S203.
Synchronizing information packet change module 506, for according to the 4th preset rules to the content in the synchronizing information packet into
Row is deleted.It can be used for executing step S103 and S203.
Specifically, as shown in figure 16, the local data update module 503 includes：
Extraction unit 5031, for extracting the data in the synchronizing information packet.It can be used for executing step S2021.
Snapshot generation unit 5032, the node data for generating other nodes according to the data in the synchronizing information packet
Snapshot.It can be used for executing step S2022.
Snapshot storage unit 5033, for the snapshot to be stored in own local.It can be used for executing step S2023.
Specifically, as shown in figure 17, the synchronizing information packet update module 504 includes：
Node data acquiring unit 5041, for obtaining the node data from node.It can be used for executing step S20210.
Communication data acquiring unit 5042, for obtaining communication data from node, from the communication data of node for from
The data received at the communication node of node, it is described from the communication node of node be the client node communicated with from node.It can
For executing step S20220.
Updating unit 5043 is used for according to the node data from node with the communication data from node to described synchronous
Packet is updated.It can be used for executing step S20230.
Specifically, the data processing module 403 in the function and host node of the data processing module 505 is almost the same,
Including：
Data capture unit 5051, for obtaining the data in the synchronizing information packet.It can be used for executing step S2031.
Snapshot generation unit 5052, the node data for generating other nodes according to the data in the synchronizing information packet
Snapshot.It can be used for executing step S2032.
Updating unit 5053, the snapshot for being stored according to the snapshot update own local.It can be used for executing step
S2033。
Local data acquiring unit 5054, for obtaining from the node data of node and being stored in from node own local
The snapshot of the node data of other nodes.It can be used for executing step S2034.
Generation unit 5055 is updated the data, for being updated the data according to the content generation got.It can be used for executing step
S2035。
Release unit 5056, for updating the data the communication node being distributed to from node by described.It can be used for executing step
S2036。
Further, the synchronizing information packet in the function and host node of synchronizing information packet change module 506 changes module 404
It is almost the same, including：
First deleting unit 5061, for deleting the node data in the synchronizing information packet from node.It can be used for executing
Step S20310.
Second deleting unit 5062, for deleting the communication data in the synchronizing information packet from node.It can be used for executing
Step S20320.
Host node described in the device of the invention embodiment is invented with embodiment of the method based in the same manner as with from node
Design.
The embodiments of the present invention also provide a kind of storage medium, the storage medium may be disposed at host node and/or from
To preserve for realizing the program code of data processing method in embodiment of the method among node.Optionally, in the present embodiment,
Above-mentioned storage medium can be located at least one of multiple network equipments of computer network network equipment.Optionally, at this
In embodiment, above-mentioned storage medium can include but is not limited to：USB flash disk, read-only memory (ROM, Read-OnlyMemory), with
Machine access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disc or CD etc. are various can to store program
The medium of code.
Specifically, Figure 18 is a kind of server architecture schematic diagram provided in an embodiment of the present invention, and the server architecture can
To be set to host node and/or among node.The server 800 can generate bigger difference because configuration or performance are different
It is different, may include one or more central processing units (central processingunits, CPU) 822 (for example, one
Or more than one processor) and memory 832, one or more store the storage medium of application programs 842 or data 844
830 (such as one or more mass memory units).Wherein, memory 832 and storage medium 830 can be of short duration storages
Or persistent storage.The program for being stored in storage medium 830 may include one or more modules (diagram is not shown), each
Module may include to the series of instructions operation in server.Further, central processing unit 822 could be provided as and deposit
Storage media 830 communicates, and the series of instructions operation in storage medium 830 is executed on server 800.Server 800 can be with
Including one or more power supplys 826, one or more wired or wireless network interfaces 850, one or more
Input/output interface 858, and/or, one or more operating systems 841, such as Windows ServerTM,
MacOSXTM, UnixTM, LinuxTM, FreeBSDTM etc..Step performed by above method embodiment can be based on the figure
Server architecture shown in 18.
The embodiment of the present invention additionally provides a kind of cluster, host node that the cluster is provided by the embodiment of the present invention and extremely
The slave node that few embodiment of the present invention is provided is constituted, and each node in the cluster further includes following modules：
Data inquiry request acquisition module obtains the data inquiry request of client node for node.It can be used for the side of execution
Method embodiment step S301.
Inquiry response generation module, the other nodes for being used to be preserved according to the node data of the node and the node
The snapshot of node data generates data query response.It can be used for executing embodiment of the method step S302.
Respond module, for data query response to be back to the client node.It can be used for the implementation of execution method
Example step S303.
It should be noted that：The embodiments of the present invention are for illustration only, can not represent the quality of embodiment.
One of ordinary skill in the art will appreciate that realizing that all or part of step of above-described embodiment can pass through hardware
It completes, relevant hardware can also be instructed to complete by program, the program can be stored in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all the present invention spirit and
Within principle, any modification, equivalent replacement, improvement and so on should all be included in the protection scope of the present invention.
Claims (15)
1. a kind of company-data processing method, which is characterized in that the data processing method includes：
Host node generates synchronizing information packet, and the synchronizing information packet is distributed in node cycle so that the synchronizing information packet
Next node is reached along the node cycle；The node cycle is made of a host node and no less than one from node, described
Node cycle includes whole nodes in the cluster；
The synchronizing information packet carries out first round transmission in the node cycle：Often reach a node after, present node according to
First preset rules change the data of own local, and are updated to synchronizing information packet according to the second preset rules,
The updated synchronizing information packet is distributed to again on node cycle and continues to transmit so that the synchronizing information packet is described in
Node cycle reaches next node；
The synchronizing information packet is obtained as starting point using the host node, executes the of the synchronizing information packet in the node cycle
Two wheels transmit：Obtain synchronizing information packet present node the data of own local are changed according to third preset rules and
It is updated the data to the publication of the communication node of the present node；The present node is according to the 4th preset rules to the synchronous letter
Content in breath packet is deleted, and the synchronizing information packet after deleting is distributed on node cycle and continues to transmit so that institute
It states synchronizing information packet and reaches next node along the node cycle；
After the host node obtains the synchronizing information packet again, the synchronizing information packet is abandoned, returns and executes the master
Node generates the step of synchronizing information packet.
2. according to the method described in claim 1, it is characterized in that, host node generation synchronizing information packet includes：
Obtain the node data of host node；
The communication data of host node is obtained, the communication data of the host node is the number received from the communication node of host node
According to the communication node of the host node is the client node communicated with host node；
Synchronizing information packet is generated according to the node data of the host node and communication data.
3. according to the method described in claim 2, it is characterized in that：
The present node according to the first preset rules to the data of own local change including：
Extract the data in the synchronizing information packet；
The snapshot of the node data of other nodes is generated according to the data in the synchronizing information packet；
The snapshot is stored in own local；
It is described according to the second preset rules to synchronizing information packet be updated including：
Obtain the node data of present node；
The communication data of present node is obtained, the communication data of the present node is to be received from the communication node of present node
The communication node of the data arrived, the present node is the client node communicated with present node；
The synchronizing information packet is updated according to the communication data of the node data of present node and the present node.
4. according to the method described in claim 3, it is characterized in that：
The present node changes the data of own local according to third preset rules and to the present node
Communication node publication update the data including：
Obtain the data in the synchronizing information packet；
The snapshot of the node data of other nodes is generated according to the data in the synchronizing information packet；
The snapshot stored according to the snapshot update own local；
It obtains the node data of present node and is stored in the snapshot of the node data of other nodes of present node own local；
It is updated the data according to the content generation got；
Described update the data is distributed to the communication node of the present node；
It is described according to the 4th preset rules to the content in the synchronizing information packet deleted including：
Delete the node data of present node in the synchronizing information packet；
Delete the communication data of present node in the synchronizing information packet.
5. according to the method described in claim 1, it is characterized in that, in the present node according to the first preset rules to itself
Further include the fast of the node data for the other nodes for deleting the storage of present node own local before local data change
According to the step of.
6. according to the method described in claim 1, it is characterized in that, if host node is deleted in cluster, according to preset node
Selection algorithm is reselecting host node from node.
7. if according to the method described in claim 1, it is characterized in that, add new slave node in cluster or delete old from section
The dynamic for causing node cycle is updated so that node cycle always including whole nodes in cluster by point.
8. according to the method described in claim 1, it is characterized in that, each node can handle the data query of client node
Request, the data processing method further include：
Node obtains the data inquiry request of client node；
The snapshot of the node data of the other nodes preserved according to the node data of the node and the node generates data and looks into
Ask response；
Data query response is back to the client node.
9. a kind of host node, which is characterized in that the host node includes：
Synchronizing information packet generation module, for generating synchronizing information packet；
Synchronizing information packet transmit control module, for by synchronizing information packet publisher node ring so that synchronizing information packet edge
The node cycle reaches next node；The node cycle is made of a host node and no less than one from node, the section
Point ring includes whole nodes in cluster；
Data processing module, for being changed to the data of own local according to third preset rules and to the host node
Communication node publication update the data；
Synchronizing information packet changes module, for being deleted the content in the synchronizing information packet according to the 4th preset rules；
Synchronizing information packet discard module, for abandoning the synchronizing information packet after the second wheel transmits in synchronizing information packet.
10. host node according to claim 9, which is characterized in that the synchronizing information packet generation module includes：
First data capture unit, the node data for obtaining host node；
The communication data of second data capture unit, the communication data for obtaining host node, the host node is from host node
Communication node at the data that receive, the communication node of the host node is the client node communicated with host node；
Synchronizing information packet generation unit, for generating synchronizing information packet according to the node data and communication data of the host node.
11. a kind of from node, which is characterized in that the node includes：
Synchronizing information packet acquisition module, for obtaining synchronizing information packet；
Synchronizing information packet transmits control module, for synchronizing information packet being distributed in node cycle so that the synchronizing information packet
Next node is reached along the node cycle；The node cycle is made of a host node and no less than one from node, described
Node cycle includes whole nodes in cluster；
Local data update module, for being changed to the data of own local according to the first preset rules；
Synchronizing information packet update module, for being updated to synchronizing information packet according to the second preset rules；
Data processing module, for being changed to the data of own local according to third preset rules and to described from node
Communication node publication update the data；
Synchronizing information packet changes module, for being deleted the content in the synchronizing information packet according to the 4th preset rules.
12. according to claim 11 from node, which is characterized in that the local data update module includes：
Extraction unit, for extracting the data in the synchronizing information packet；
Snapshot generation unit, the snapshot of the node data for generating other nodes according to the data in the synchronizing information packet；
Snapshot storage unit, for the snapshot to be stored in own local.
13. according to claim 11 from node, which is characterized in that the synchronizing information packet update module includes：
Node data acquiring unit, for obtaining the node data from node；
Communication data acquiring unit is in leading to from node from the communication data of node for obtaining the communication data from node
The data that receive at letter node, it is described from the communication node of node be the client node communicated with from node；
Updating unit, for according to from the node data of node and the communication data from node to the synchronizing information packet into
Row update.
14. a kind of cluster, the cluster includes any one in host node and claim 11-13 described in claim 9 or 10
Slave node described in.
15. cluster according to claim 14, which is characterized in that the host node of the cluster and include from node each
Following modules：
Data inquiry request acquisition module obtains the data inquiry request of client node for node；
Inquiry response generation module, the node of other nodes for being preserved according to the node data of the node and the node
The snapshot of data generates data query response；
Respond module, for data query response to be back to the client node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710357042.4A CN107277111B (en) | 2017-05-19 | 2017-05-19 | A kind of company-data processing method, master node, slave node and cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710357042.4A CN107277111B (en) | 2017-05-19 | 2017-05-19 | A kind of company-data processing method, master node, slave node and cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107277111A CN107277111A (en) | 2017-10-20 |
| CN107277111B true CN107277111B (en) | 2018-09-04 |
Family
ID=60065405
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710357042.4A Active CN107277111B (en) | 2017-05-19 | 2017-05-19 | A kind of company-data processing method, master node, slave node and cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107277111B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112054926B (en) * | 2020-08-31 | 2023-03-10 | 深圳前海微众银行股份有限公司 | Cluster management method and device, electronic equipment and storage medium |
| CN112804337A (en) * | 2021-01-22 | 2021-05-14 | 苏州浪潮智能科技有限公司 | Main node pressure allocation method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101178677A (en) * | 2007-11-09 | 2008-05-14 | 中国科学院计算技术研究所 | Computer system for protecting software and method for protecting software |
| CN102737098A (en) * | 2011-03-29 | 2012-10-17 | 日本电气株式会社 | Distributed file system |
| CN102929745A (en) * | 2012-09-26 | 2013-02-13 | 东软集团股份有限公司 | Status data updating method and device for virtual equipments |
| CN105069152A (en) * | 2015-08-25 | 2015-11-18 | 航天恒星科技有限公司 | Data processing method and apparatus |
| CN105554135A (en) * | 2015-12-23 | 2016-05-04 | 北京奇虎科技有限公司 | Method and device for synchronizing meta information among digital nodes |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8856281B2 (en) * | 2010-03-22 | 2014-10-07 | At&T Intellectual Property I, L.P. | Internet protocol version 6 content routing |
| CN105095486A (en) * | 2015-08-17 | 2015-11-25 | 浪潮(北京)电子信息产业有限公司 | Cluster database disaster recovery method and device |
-
2017
- 2017-05-19 CN CN201710357042.4A patent/CN107277111B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101178677A (en) * | 2007-11-09 | 2008-05-14 | 中国科学院计算技术研究所 | Computer system for protecting software and method for protecting software |
| CN102737098A (en) * | 2011-03-29 | 2012-10-17 | 日本电气株式会社 | Distributed file system |
| CN102929745A (en) * | 2012-09-26 | 2013-02-13 | 东软集团股份有限公司 | Status data updating method and device for virtual equipments |
| CN105069152A (en) * | 2015-08-25 | 2015-11-18 | 航天恒星科技有限公司 | Data processing method and apparatus |
| CN105554135A (en) * | 2015-12-23 | 2016-05-04 | 北京奇虎科技有限公司 | Method and device for synchronizing meta information among digital nodes |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107277111A (en) | 2017-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102544628B1 (en) | System for a data sharing platform in a block chain based distributed data sharing environment, method for searching data index in the system and method for providing seartch index in the system | |
| Montresor et al. | Distributed k-core decomposition | |
| CN107086907B (en) | Key synchronization and packaging transfer method and device for quantum key distribution process | |
| CN112712182B (en) | Model training method and device based on federal learning and storage medium | |
| CN103544303B (en) | A kind of method of data synchronization, system and equipment | |
| CN107277111B (en) | A kind of company-data processing method, master node, slave node and cluster | |
| CN105518641B (en) | Point-to-Point Data clone method, equipment and system and host node switching method, equipment and system | |
| CN105991720B (en) | Configuration change method, equipment and system | |
| CN101986272A (en) | Task scheduling method under cloud computing environment | |
| CN105264930B (en) | Sending node and its reporting cached state method | |
| CN104239555B (en) | Parallel data mining system and its implementation based on MPP | |
| CN105208058B (en) | The information interaction system shared based on web sessions | |
| CN110196761A (en) | Delay task processing method and processing device | |
| Langer et al. | MPCA SGD—a method for distributed training of deep learning models on spark | |
| CN107231400A (en) | The synchronous method and device of a kind of data | |
| CN111090519B (en) | Task execution method and device, storage medium and electronic equipment | |
| CN110083504B (en) | Running state monitoring method and device for distributed tasks | |
| CN105471893B (en) | A kind of distributed equivalent data flow connection method | |
| CN106452899B (en) | A kind of distributed data digging system and method | |
| CN108023968A (en) | A kind of session information synchronous method, device and equipment | |
| CN104468722A (en) | Method for classified storage of training data in navigation management training system | |
| CN105069029A (en) | Real-time ETL (extraction-transformation-loading) system and method | |
| CN110602118A (en) | Virtualization data remote encryption security system and method | |
| CN108418871A (en) | A kind of cloud storage performance optimization method and system | |
| CN107633026A (en) | Data syn-chronization abnormality eliminating method, device and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |