CN113596746B - Cluster message processing method and device, electronic equipment and medium - Google Patents
Cluster message processing method and device, electronic equipment and medium Download PDFInfo
- Publication number
- CN113596746B CN113596746B CN202110831096.6A CN202110831096A CN113596746B CN 113596746 B CN113596746 B CN 113596746B CN 202110831096 A CN202110831096 A CN 202110831096A CN 113596746 B CN113596746 B CN 113596746B
- Authority
- CN
- China
- Prior art keywords
- node
- service
- nodes
- cluster
- communication
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/06—Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services
- H04W4/10—Push-to-Talk [PTT] or Push-On-Call services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/06—Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The disclosure relates to a method, an apparatus, an electronic device and a medium for processing a cluster message; wherein, the method comprises the following steps: acquiring attribute information respectively corresponding to service nodes in a cluster of a data structure server; dividing service nodes in the cluster according to the attribute information to obtain at least two node sets; determining communication nodes respectively corresponding to at least two node sets, wherein the communication node corresponding to each node set is a service node in the node sets; if the node service message generated in the cluster is monitored, the node service message is broadcasted to the communication node, so that the communication node broadcasts the node service message to other service nodes in a node set where the communication node is located, and the node service message is used for indicating at least one of node addition, node deletion and communication node change of the service node. The embodiment can improve the transmission efficiency of the cluster message.
Description
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a method and an apparatus for processing a cluster message, an electronic device, and a medium.
Background
A REmote DIctionary Server (redis) is a data structure Server using in-memory storage that can be used as a database, cache, and message queue agent. Efficient storage of data storage types such as character strings, hash tables, lists, sets, ordered sets, and bitmaps can be supported in redis.
However, a conventional redis cluster constructed by a redis may include a plurality of service nodes, which are used to acquire and store a cluster message (such as an additional node or a deletion node), and each service node in the redis cluster performs message transmission in a point-to-point manner. If a new service node is added to the redis cluster, the message of adding the new service node needs to be transmitted to all service nodes in the redis cluster respectively.
Therefore, when the number of service nodes in the redis cluster is huge, the transmission efficiency of the cluster message is greatly reduced by a point-to-point message transmission mode.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, the present disclosure provides a method, an apparatus, an electronic device, and a medium for processing a group message.
In a first aspect, this embodiment provides a method for processing a cluster message, including:
acquiring attribute information respectively corresponding to service nodes in a cluster of a data structure server;
dividing service nodes in the cluster according to the attribute information to obtain at least two node sets;
determining communication nodes respectively corresponding to the at least two node sets, wherein the communication node corresponding to each node set is a service node in the node set;
if the node service message is generated in the cluster, the node service message is broadcasted to the communication node, so that the communication node broadcasts the node service message to other service nodes in a node set where the communication node is located, and the node service message is used for indicating at least one of node addition, node deletion and communication node change of the service node.
Optionally, the determining the communication nodes respectively corresponding to the at least two node sets includes:
acquiring data storage quantity respectively corresponding to service nodes in each node set;
and taking the service node corresponding to the minimum data storage amount as a communication node of the node set.
Optionally, the determining the communication nodes respectively corresponding to the at least two node sets includes:
acquiring performance parameter values respectively corresponding to service nodes in each node set;
and selecting the communication node according to the performance parameter value.
Optionally, the node service message is generated when the task throughput of the first node set in the cluster is greater than a preset first throughput threshold;
before broadcasting the node service message to the corresponding node, the method further includes:
adding a first preset number of service nodes into the first set of nodes;
the broadcasting the node service message to the corresponding node comprises:
broadcasting the identification of the added serving node and the belonging information of the first set of nodes to other sets of nodes in the cluster.
Optionally, the node service message is generated when the task throughput of a second node set in the cluster is less than or equal to a preset second throughput threshold;
before broadcasting the node service message to the corresponding node, the method further includes:
deleting a second preset number of service nodes from the second node set;
the broadcasting the node service message to the corresponding node includes:
broadcasting the identity of the deleted service node and the belonging information of the second set of nodes to other sets of nodes in the cluster.
Optionally, the node service message is generated when a working state of a corresponding node of a third node set in the cluster is abnormal;
before broadcasting the node service message to the corresponding node, the method further includes:
changing a service node in the third node set into a new communication node;
the broadcasting the node service message to the corresponding node includes:
and broadcasting the identification of the new communication node and the belonging information of the third node set to other node sets in the cluster.
Optionally, the method further includes: and acquiring and recording the node states of the added service nodes in the first node set.
Optionally, the method further includes: and acquiring and recording the node state of the new communication node in the third node set.
Optionally, the attribute information includes location information; the dividing the service nodes in the cluster according to the position information to obtain at least two node sets comprises:
selecting at least two service nodes from the cluster as target nodes;
according to the position information respectively corresponding to the service nodes, when the distance between the target node and other service nodes is determined to be smaller than a preset distance threshold value, the target node and other service nodes are divided into a node set;
if the fourth service node is not divided, dividing the fourth service node into a node set closest to the fourth service node;
and carrying out node duplicate removal operation on all the node sets obtained by division to obtain at least two node sets, wherein the node duplicate removal operation comprises deleting the service nodes repeatedly divided in the node sets, and the service nodes contained in each of the at least two node sets are different.
In a second aspect, this embodiment provides a device for processing a cluster message, including:
the acquisition module is used for acquiring attribute information respectively corresponding to service nodes in a cluster of the data structure server;
the dividing module is used for dividing the service nodes in the cluster according to the attribute information to obtain at least two node sets;
a determining module, configured to determine communication nodes corresponding to the at least two node sets, where a communication node corresponding to each node set is a service node in the node set;
and the broadcasting module is used for broadcasting the node service message to the communication node if the node service message generated in the cluster is monitored, so that the communication node broadcasts the node service message to other service nodes in a node set where the communication node is located, and the node service message is used for indicating at least one of node addition, node deletion and communication node change of the service node.
Optionally, the determining module is specifically configured to:
acquiring data storage quantity respectively corresponding to service nodes in each node set;
and taking the service node corresponding to the minimum data storage amount as a communication node of the node set.
Optionally, the determining module is specifically configured to:
acquiring performance parameter values respectively corresponding to service nodes in each node set;
and selecting the communication node according to the performance parameter value.
Optionally, the node service message is generated when the task throughput of the first node set in the cluster is greater than a preset first throughput threshold;
further comprising: adding a module;
an adding module configured to add a first preset number of service nodes to the first node set;
the broadcast module is specifically configured to:
broadcasting the identification of the added serving node and the affiliation information of the first set of nodes to other sets of nodes in the cluster.
Optionally, the node service message is generated when the task throughput of a second node set in the cluster is less than or equal to a preset second throughput threshold;
further comprising: a deletion module;
a deleting module, configured to delete a second preset number of service nodes from the second node set;
the broadcast module is specifically configured to:
broadcasting the identity of the deleted service node and the belonging information of the second set of nodes to other sets of nodes in the cluster.
Optionally, the node service message is generated when a working state of a corresponding node of a third node set in the cluster is abnormal;
further comprising: a change module;
a changing module, configured to change a service node in the third node set to a new correspondent node;
a broadcast module specifically configured to:
and broadcasting the identification of the new communication node and the belonging information of the third node set to other node sets in the cluster.
Optionally, the obtaining module is further configured to obtain and record node states of the added service nodes in the first node set.
Optionally, the obtaining module is further configured to obtain and record a node state of the new communication node in the third node set.
Optionally, the attribute information includes location information;
a partitioning module specifically configured to:
selecting at least two service nodes from the cluster as target nodes;
according to the position information respectively corresponding to the service nodes, when the distance between the target node and other service nodes is determined to be smaller than a preset distance threshold value, the target node and other service nodes are divided into a node set;
if the fourth service node is not divided, dividing the fourth service node into a node set closest to the fourth service node;
and carrying out node duplicate removal operation on all the node sets obtained by division to obtain at least two node sets, wherein the node duplicate removal operation comprises deleting the service nodes repeatedly divided in the node sets, and the service nodes contained in each of the at least two node sets are different.
In a third aspect, the present embodiment further provides an electronic device, including:
one or more processors;
a storage device to store one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the method for processing the cluster message in any of the embodiments of the present invention.
In a fourth aspect, this embodiment further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is used to implement the method for processing a cluster message in any one of the embodiments of the present invention when executed by a processor.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages: the service nodes in the cluster can be grouped, the communication nodes in each node set are used for receiving the node service messages generated in the cluster, and the number of the service nodes for receiving the messages is greatly reduced, so that the service nodes in each node set can realize the parallel receiving of the messages based on the communication nodes, and the transmission efficiency of the cluster messages is greatly improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a method for processing a cluster message according to this embodiment;
fig. 2 is a schematic diagram illustrating a group message provided in this embodiment;
fig. 3 is a schematic diagram illustrating another group message provided in this embodiment;
fig. 4 is a schematic flowchart of another cluster message processing method provided in this embodiment;
fig. 5 is a schematic flowchart of another method for processing a cluster message according to this embodiment;
fig. 6 is a schematic flowchart of another cluster message processing method provided in this embodiment;
fig. 7 is a schematic structural diagram of another cluster message processing apparatus provided in this embodiment;
fig. 8 is a schematic structural diagram of an electronic device provided in this embodiment.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
A redis cluster is a cluster comprised of a plurality of service nodes, wherein the service nodes may include a master node and a slave node. Each service node in the redis cluster needs to be connected with other service nodes (master nodes or slave nodes), that is, the service nodes in the redis cluster are connected pairwise to perform the transfer of the cluster message.
The slot information in each slot can be fully covered by all nodes in the redis cluster, and therefore all the keys can be mapped to the service nodes corresponding to the slots in the redis cluster.
The information in each slot includes key value data, where key represents a keyword and value represents information corresponding to the keyword, that is, the key and the value in each slot are in a one-to-one correspondence relationship. If the key is name, the value is Zhang III.
Fig. 1 is a schematic flowchart of a method for processing a cluster message provided in this embodiment. The present embodiment may be applicable to the case where a node service message is broadcast in a cluster. The method of the embodiment can be executed by a processing device of the cluster message, the device can be realized by adopting a hardware/software mode, and can be configured in the electronic equipment; wherein the electronic device may comprise a server. The method for processing the cluster message in any embodiment of the application can be realized. As shown in fig. 1, the method specifically includes the following steps:
s110, obtaining attribute information corresponding to the service nodes in the cluster of the data structure server.
In this embodiment, the cluster of the data structure server is composed of a plurality of service nodes, and each service node has its corresponding attribute information. The attribute information may include, but is not limited to: the system comprises position information, data information and storage information, wherein the position information corresponding to one service node is an identifier of a physical position where the service node is located, such as a node coordinate, and the storage information corresponding to one service node is an identifier of data stored in the service node, such as a data number, a data name and the like.
The node coordinates can be two-dimensional coordinates or three-dimensional coordinates, so that the position of each service node can be accurately positioned according to the position information corresponding to each service node, and a plurality of service nodes in the cluster can be uniformly managed.
And S120, dividing the service nodes in the cluster according to the attribute information to obtain at least two node sets.
In this embodiment, when the attribute information is location information, the location information corresponding to each service node may be different, that is, a plurality of service nodes in the cluster may be distributed at different locations, and when the cluster notifies each service node of the cluster message, the cluster message needs to be broadcast to each service node, so that all the service nodes can know the dynamic state in the cluster.
For example, if a service node a is newly added to the cluster, the service node B in the cluster is required to transmit a message that the service node a joins the cluster to other service nodes in the cluster, respectively, and a service node that may be farther from the service node B takes a longer time to acquire the message. Thus, when the number of nodes in the cluster is large, the message propagation in the cluster is low.
In this embodiment, two or more service nodes with a short distance may be divided into one node set, and one node set is regarded as a group of nodes, that is, a cluster may include a small number of node sets, and the time for the service nodes in each node set to acquire messages may be synchronized.
In addition, when the attribute information is data information, two or more service nodes with higher data information similarity can be divided into one node set, one node set can be regarded as a group of nodes, that is, a cluster can contain a smaller number of node sets, and the time for the service nodes in each node set to acquire messages can be synchronized.
S130, communication nodes corresponding to at least two node sets are determined, wherein the communication node corresponding to each node set is a service node in the node sets.
In this embodiment, after dividing a plurality of service nodes in a cluster into at least two node sets, the service nodes in each node set do not need to perform one-to-one communication, and the authority of message communication between the node sets can be given to one service node for execution, and one communication node can be selected from each node set to perform communication connection with communication nodes in other node sets, thereby avoiding the problem of non-uniformity of message reception of the service nodes caused by network delay.
The specific implementation manner determined by the communication node in one node set may be implemented by different selection policies, where the selection policies include, but are not limited to, an identification number of a slot of a service node, or physical performance of the service node, or data volume held by the service node, or the number of slots held by the service node.
Based on the description of the above embodiments, this embodiment provides a comparison between an original structural schematic diagram of a service node in a cluster and a structural schematic diagram of a service node in a cluster of this embodiment, which can be seen in fig. 2 and fig. 3. Fig. 2 is an original cluster message graph, and fig. 3 is a cluster message graph obtained by dividing according to the method of the present embodiment.
In fig. 2, it is assumed that 6 service nodes, respectively M1, M2, M3, S1, S2 and S3, are included in the original cluster, and each service node performs bidirectional communication, that is, M1 is bidirectionally connected to M2, M3, S1, S2 and S3, M2 is bidirectionally connected to M1, M3, S1, S2 and S3, M3 is bidirectionally connected to M1, M2, S1, S2 and S3, S1 is bidirectionally connected to M2, M3, S2 and S3, S2 is bidirectionally connected to M1, M2, M3, S1 and S3, and S3 is bidirectionally connected to M1, M2, M3, S1 and S2.
After the service nodes in the cluster are divided by the method of the embodiment, a plurality of node sets are obtained. In fig. 3, the service nodes in fig. 2 are divided according to the location information of each service node to obtain 3 node sets, which are respectively a node set composed of S1 and M1, a node set composed of S2 and M2, and a node set composed of S3 and M3. And the service nodes in each node set are connected with each other, only the communication nodes in the node sets are connected with each other, if the communication node in the node set composed of S1 and M1 is M1, the communication node in the node set composed of S2 and M2 is M2, and the communication node in the node set composed of S3 and M3 is M3, then M1 is respectively connected with M2 and M3, M2 is respectively connected with M1 and M3, and M3 is respectively connected with M1 and M2.
Therefore, based on the conversion from fig. 2 to fig. 3, it can be seen intuitively that the present embodiment can simplify the connection relationship between the service nodes in the cluster without changing the original cluster, and meanwhile, the service node in each node set can implement message communication with the service node (or the communication node) in other node sets through the communication node.
And S140, if the node service message generated in the cluster is monitored, broadcasting the node service message to the communication node, so that the communication node broadcasts the node service message to other service nodes in the node set where the communication node is located.
The node service message is used for indicating at least one of node addition, node deletion and communication node change of the service node.
In this embodiment, the node service message is a change message of service nodes in the cluster, such as adding a node in a certain node set, or subtracting a node in a certain node set, or replacing a service node in a certain node set with a correspondent node or changing a correspondent node into a service node.
The cluster server monitors each node set included in the cluster in real time, and when detecting that a node service message occurs in the cluster, the cluster server needs to transmit the node service message to service nodes included in each node set in the cluster. The present embodiment may deliver the node service message to the corresponding node in each node set to instruct the corresponding node to deliver the node service message to each service node in the node set at the same time.
Therefore, each service node in the node set can receive the node service message in time, the probability of message receiving delay caused by network delay is reduced, and the transmission efficiency of the cluster message is effectively ensured.
It should be noted that, in each node set, the corresponding node may broadcast the node service message to other service nodes, and the other service nodes also have communication connection therebetween, and may also perform message intercommunication, and meanwhile, the other service nodes may also transmit the message to the corresponding node in the node set. Therefore, all service nodes in one node set can conveniently carry out message communication, and the message synchronism of the service nodes in the node set is improved.
The service nodes in the cluster can be grouped, the communication nodes in each node set are used for receiving the node service messages generated in the cluster, and the number of the service nodes for receiving the messages is greatly reduced, so that the service nodes in each node set can realize the parallel receiving of the messages based on the communication nodes, and the transmission efficiency of the cluster messages is greatly improved.
Based on the description of the above embodiments, the specific determination method of the communication node in one node set has multiple implementation manners, which is not limited in this embodiment.
In this embodiment, optionally, determining the communication nodes corresponding to the at least two node sets respectively includes:
acquiring data storage quantity respectively corresponding to service nodes in each node set;
and taking the service node corresponding to the minimum data storage amount as a communication node of the node set.
In this embodiment, one service node may correspond to one or more slot slots in the cluster, data storage amounts of key value data included in each slot are different, and the data storage amount held by one service node can effectively reflect resource consumption of the cluster occupied by the service node.
When a communication node of a node set is determined, counting the data storage quantity held by each service node in the node set, comparing all the data storage quantities, and selecting the service node corresponding to the minimum data storage quantity obtained in the comparison result as the communication node of the node set, so that the communication node has the authority of message transmission with the communication nodes of other node sets.
Therefore, the service node with the least data storage capacity is selected as the communication node in the node set, so that the resource consumption of the cluster occupied during message transmission can be effectively reduced, and the message transmission efficiency is further improved.
In this embodiment, optionally, determining the communication nodes corresponding to the at least two node sets respectively includes:
acquiring performance parameter values respectively corresponding to service nodes in each node set;
and selecting the communication node according to the performance parameter value.
When a communication node is selected from a node set, the service node corresponding to the optimal performance parameter value can be used as the communication node of the node set.
In this embodiment, the performance parameter value is a quantitative evaluation value of the physical performance of a service node, and is used to measure the quality of the physical performance of a service node.
It should be noted that the optimal performance parameter can be divided into a minimum performance parameter and a maximum performance parameter according to different performance parameters.
In some embodiments, the performance parameters may include: thread count, cache occupancy, CPU occupancy. The smaller the performance parameter value of the performance parameter is, the more available the service node is currently, and the service node corresponding to the minimum performance parameter value can be selected as the communication node of the node set, so that the speed during message transmission can be effectively ensured, and the message transmission speed between the communication nodes is further improved.
In other embodiments, the performance parameters may include: node idleness and resource storage space. At this time, the service node corresponding to the maximum performance parameter can be selected as the communication node of the node set, so as to ensure the message transmission efficiency.
In addition, determining the corresponding communication nodes of at least two node sets may further include the following embodiments.
In some embodiments, the numerical value of the identification number of the slot corresponding to the service node in each node set is obtained; and taking the service node corresponding to the identification number with the minimum numerical value as a communication node in the node set.
In other embodiments, the number of slot slots respectively corresponding to the service nodes in each node set is obtained; and taking the service node corresponding to the lot slot with the minimum number as the communication node in the node set. Therefore, the resource loss of the message transmitted between the communication nodes can be effectively reduced.
In this embodiment, optionally, the attribute information includes location information; dividing service nodes in the cluster according to the attribute information to obtain at least two node sets, including:
selecting at least two service nodes from the cluster as target nodes;
according to the position information respectively corresponding to the service nodes, when the distance between the target node and other service nodes is determined to be smaller than a preset distance threshold value, the target node and other service nodes are divided into a node set;
if the fourth service node is detected not to be divided, dividing the fourth service node into a node set with the nearest distance;
and carrying out node duplicate removal operation on all the node sets obtained by division to obtain at least two node sets, wherein the node duplicate removal operation comprises deleting the service nodes repeatedly divided in the node sets, and the service nodes contained in each of the at least two node sets are different.
In this embodiment, the plurality of service nodes may be divided according to the physical distance between each two service nodes in the cluster, so as to reduce the network delay during message transmission.
When the target node is determined, at least two service nodes can be selected from the cluster as the target node according to the position information of the service nodes in the cluster, and if two service nodes with a distance greater than a certain distance threshold value in the cluster can be used as the target node, the distance threshold value can be adjusted and set according to the position information of the service nodes in the cluster.
Or, when determining the target node, at least two service nodes may be selected from the cluster as the target node according to the data amount included in the service nodes in the cluster, for example, a service node with a data amount of a first data amount and a service node with a data amount of a second data amount in the cluster may be used as the target node. The numerical values of the first data volume and the second data volume have a certain difference, so that each service node in the cluster is effectively divided.
However, if there may be a case where one or more service nodes in the cluster are not yet divided, node supplementation needs to be performed on the initial node set, and meanwhile, if there may be a case where nodes are duplicated in the initial node set, node duplication needs to be performed on the initial node set, so as to obtain a finally divided node set.
The fourth service node is a service node which is not divided in the cluster, and the fourth service node can be divided into a node set where a target node with the smallest distance is located according to the distance between the fourth service node and the plurality of target nodes.
And after the node supplementation of the node set is completed, carrying out node duplication removal operation on the node set. And detecting whether the repeated nodes exist in all the node sets, if so, deleting the repeated nodes from any node set with the repeated nodes, so that only one node set keeps the repeated nodes. Or selecting a repeat node in a node set which is closest to the target node according to the distance between the repeat node and the target node in the node set with the repeat node, and deleting the repeat node in other node sets with the repeat node.
Therefore, all service nodes in the cluster are guaranteed to be divided into one node set, and the service nodes contained in each node set are not repeated.
Fig. 4 is a schematic flowchart of another method for processing a cluster message according to this embodiment. The present embodiment is based on the above embodiment, wherein the node service message is generated when the task throughput of the first node set in the cluster is greater than the preset first throughput threshold. Before S140, the method of this embodiment may further include:
s131, adding a first preset number of service nodes into the first node set.
In this embodiment, the first node set is one node set of a plurality of node sets in the cluster. When it is detected that the task processing amount of the first node set is greater than a preset first processing amount threshold, it indicates that the service node in the first node set is difficult to timely and effectively process the task in the first node set, and node expansion needs to be performed on the first node set to meet the task processing requirement in the first node set.
The first preset number of service nodes may not be a service node included in any node set in the cluster, but is a new service node outside the cluster.
It should be noted that the first preset number of service nodes may also be one or more service nodes in a certain node set in the cluster, so that node scheduling is performed in the cluster to meet the node expansion requirement of the first node set.
Further, one possible implementation of S140 is as follows:
s1401, the identification of the added service node and the belonging information of the first node set are broadcasted to other node sets in the cluster.
In this embodiment, the identifier of the added service node may be an identity number of the service node in the first node set, or a device name or a device identifier of the service node.
The identifier of the added service node and the information to which the first node set belongs are included information of the added service node and the first node set, that is, the added service node currently belongs to the first node set. Therefore, the added service node and the relationship of the first node set are effectively transmitted to other service nodes.
In this embodiment, optionally, the method of this embodiment further includes:
and acquiring and recording the node state of the added service node in the first node set.
In this embodiment, regarding the added service node as the first node, the node state of the first node in the first node set may include: newly establishing connection, normal state, failure judgment and failure declaration.
Wherein the newly created connection may indicate that the first node has joined the first set of nodes but has not yet been assigned a task. The normal state may indicate that the first node joined the first set of nodes and has already been assigned a task. The failure of judgment may indicate that the first node is abnormal in the process of joining the first node set, and the first node set cannot be joined for a while. The declaration of the failure may indicate that the first node failed to join the first set of nodes and failed to join the first set of nodes.
Therefore, by recording the node state of the newly added service node in the first node set, the newly added service node can be comprehensively known in real time.
Fig. 5 is a schematic flowchart of another method for processing a cluster message according to this embodiment. The present embodiment is based on the above embodiment, wherein the node service message is generated when the task throughput of the second node set in the cluster is less than or equal to the preset second throughput threshold. Before S140, the method of this embodiment may further include:
s132, deleting a second preset number of service nodes from the second node set.
And the deleted service node is one service node in the second node set.
In this embodiment, the second node set is one node set of a plurality of node sets in the cluster. When it is detected that the task throughput of the second node set is less than or equal to the preset second throughput threshold, it indicates that the service node in the second node set can timely and effectively process the task in the second node set.
After deleting a second preset number of service nodes from the second node set, the deleted nodes can be completely deleted from the cluster. Or adding the deleted node to other node sets with larger task quantity requirements in the cluster to meet the task quantity requirements of the node sets.
It should be noted that the first throughput threshold is not related to the second throughput threshold, and may be set effectively according to the task amounts of the first node set and the second node set.
Further, one possible implementation of S140 is as follows:
s1402, broadcasting the identification of the deleted service node and the belonging information of the second node set to other node sets in the cluster.
In this embodiment, the identifier of the deleted service node may be an identity number of the service node in the second node set, or a device name or a device identifier of the service node.
The deleted service node identifier and the information to which the second node set belongs are the information included in the service node and the second node set, that is, the service node does not belong to the second node set at present.
Therefore, the relationship between the deleted service node and the second node set is effectively transmitted to other service nodes. So that other service nodes can timely know the change message in the node set with node change.
Fig. 6 is a flowchart illustrating a processing method of a cluster message according to this embodiment. The present embodiment is based on the above embodiment, wherein the node service message is generated when the working state of the corresponding node of the third node set in the cluster is abnormal. Before S140, the method of this embodiment may further include:
s133, changing a service node in the third node set into a new communication node.
In this embodiment, the third node set is one node set of a plurality of node sets in the cluster. When the working state of the communication node of the third node set is detected to be abnormal, the communication node is indicated to be difficult to effectively continue to transmit messages with other communication nodes, and in order to ensure the message acquisition timeliness of the third node set, the communication node of the third node set needs to be changed.
The selection of the new communication node in the third node set can select one service node as the new communication node according to the performance parameter value of each service node in the third node set, thereby ensuring that the third node set can continue to effectively transmit messages by using the new communication node.
Further, one possible implementation of S140 is as follows:
and S1403, broadcasting the identification of the new communication node and the belonging information of the third node set to other node sets in the cluster.
In this embodiment, the identifier of the new corresponding node may be an identity number of the corresponding node in the third node set, or a device name or a device identifier of the corresponding node.
The identifier of the new correspondent node and the information to which the third node set belongs are identity information of the service node in the third node set, that is, the service node is currently a correspondent node of the third node set.
Therefore, the relationship between the new communication node and the third node set is effectively transmitted to other service nodes. And other service nodes can timely know the message of the change of the communication node in the third node set.
In this embodiment, optionally, the method of this embodiment further includes:
and acquiring and recording the node state of the new communication node in the third node set.
In this embodiment, the node state of the third node in the third node set may include a write state or a read state.
The write state may indicate that the third node is capable of transmitting the message acquired by the other corresponding node to other service nodes in the third node set. The read status indicates that the third node can only read messages communicated by other serving nodes in the third set of nodes.
Therefore, the node state of the third node in the third node set is recorded, and the third node can be comprehensively monitored in real time.
Fig. 7 is a schematic structural diagram of a device for processing a cluster message provided in this embodiment; the device is configured in the electronic device, and can implement the method for processing the cluster message in any embodiment of the present application. The device specifically comprises the following steps:
an obtaining module 710, configured to obtain attribute information corresponding to service nodes in a cluster of a data structure server;
a dividing module 720, configured to divide the service nodes in the cluster according to the attribute information to obtain at least two node sets;
a determining module 730, configured to determine communication nodes corresponding to the at least two node sets, where a communication node corresponding to each node set is a service node in the node set;
the broadcasting module 740 is configured to broadcast the node service message to the corresponding node if it is monitored that the node service message is generated in the cluster, so that the corresponding node broadcasts the node service message to other service nodes in a node set where the corresponding node is located, where the node service message is used to indicate at least one of node addition, node deletion, and communication node change to the service node.
In this embodiment, optionally, the determining module 730 is specifically configured to:
acquiring data storage quantity respectively corresponding to service nodes in each node set;
and taking the service node corresponding to the minimum data storage amount as a communication node of the node set.
In this embodiment, optionally, the determining module 730 is specifically configured to:
acquiring performance parameter values respectively corresponding to service nodes in each node set;
and selecting the communication node according to the performance parameter value.
In this embodiment, optionally, the node service message is generated when the task throughput of the first node set in the cluster is greater than a preset first throughput threshold; the device of the embodiment further comprises: adding a module;
an adding module configured to add a first preset number of service nodes to the first node set;
the broadcasting module 740 is specifically configured to:
broadcasting the identification of the added serving node and the belonging information of the first set of nodes to other sets of nodes in the cluster.
In this embodiment, optionally, the node service message is generated when the task throughput of the second node set in the cluster is less than or equal to a preset second throughput threshold; the device of the embodiment further comprises: deleting the module;
a deleting module, configured to delete a second preset number of service nodes from the second node set;
the broadcasting module 740 is specifically configured to:
broadcasting the identity of the deleted service node and the belonging information of the second set of nodes to other sets of nodes in the cluster.
In this embodiment, optionally, the node service message is generated when the working state of the corresponding node of the third node set in the cluster is abnormal; the device of the embodiment further comprises: further comprising: a change module;
a changing module, configured to change a service node in the third node set to a new correspondent node;
the broadcasting module 740 is specifically configured to:
and broadcasting the identification of the new communication node and the belonging information of the third node set to other node sets in the cluster.
In this embodiment, optionally, the obtaining module 710 is further configured to obtain and record a node state of the added service node in the first node set.
In this embodiment, optionally, the obtaining module 710 is further configured to obtain and record a node state of the new communication node in the third node set.
In this embodiment, optionally, the attribute information includes location information; the dividing module 720 is specifically configured to:
selecting at least two service nodes from the cluster as target nodes;
according to the position information respectively corresponding to the service nodes, when the distance between the target node and other service nodes is determined to be smaller than a preset distance threshold value, the target node and other service nodes are divided into a node set;
if the fourth service node is detected not to be divided, dividing the fourth service node into a node set with the nearest distance;
and carrying out node duplicate removal operation on all the node sets obtained by division to obtain at least two node sets, wherein the node duplicate removal operation comprises deleting the service nodes repeatedly divided in the node sets, and the service nodes contained in each of the at least two node sets are different.
By the cluster message processing device, the service nodes in the cluster can be grouped, the communication nodes in each node set are used for receiving the node service messages generated in the cluster, and the number of the service nodes for receiving the messages is greatly reduced, so that the service nodes in each node set can realize the parallel receiving of the messages based on the communication nodes, and the transmission efficiency of the cluster messages is greatly improved.
The processing device for the cluster message provided by the embodiment of the invention can execute the processing method for the cluster message provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 8 is a schematic structural diagram of an electronic device provided in this embodiment. As shown in fig. 8, the electronic device includes a processor 810, a memory 820, an input device 830, and an output device 840; the number of the processors 810 in the electronic device may be one or more, and one processor 810 is taken as an example in fig. 8; the processor 810, the memory 820, the input device 830 and the output device 840 in the electronic apparatus may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 8.
The memory 820 is a computer-readable storage medium and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the cluster message processing method in the embodiment of the present invention. The processor 810 executes various functional applications and data processing of the electronic device by executing software programs, instructions and modules stored in the memory 820, that is, the method for processing the cluster message provided by the embodiment of the present invention is implemented.
The memory 820 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 820 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 820 can further include memory located remotely from the processor 810, which can be connected to electronic devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 830 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus, and may include a keyboard, a mouse, and the like. The output device 840 may include a display device such as a display screen.
The present embodiment also provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are used to implement the processing method of cluster messages provided by the embodiments of the present invention.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the method for processing a cluster message provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the above search apparatus, each included unit and module are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.
The above description is only a specific embodiment of the present embodiment, so that those skilled in the art can understand or implement the present embodiment. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the embodiments. Thus, the present embodiments are not intended to be limited to the embodiments shown herein but are to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. A method for processing a cluster message, the method comprising:
acquiring attribute information respectively corresponding to service nodes in a cluster of a data structure server;
dividing service nodes in the cluster according to the attribute information to obtain at least two node sets;
determining communication nodes respectively corresponding to the at least two node sets, wherein the communication node corresponding to each node set is a service node in the node set, the communication node corresponding to each node set is in communication connection with communication nodes in other node sets, and the service nodes in each node set are mutually connected;
if the node service message is generated in the cluster, the node service message is broadcasted to the communication node, so that the communication node broadcasts the node service message to other service nodes in a node set where the communication node is located, and the node service message is used for indicating at least one of node addition, node deletion and communication node change of the service node.
2. The method according to claim 1, wherein the determining the corresponding communication nodes respectively corresponding to the at least two node sets comprises:
acquiring data storage quantity respectively corresponding to service nodes in each node set;
and taking the service node corresponding to the minimum data storage amount as a communication node of the node set.
3. The method according to claim 1, wherein the determining the corresponding communication nodes respectively corresponding to the at least two node sets comprises:
acquiring performance parameter values respectively corresponding to service nodes in each node set;
and selecting the communication node according to the performance parameter value.
4. The method of claim 1, wherein the node service message is generated when a task throughput of a first set of nodes in the cluster is greater than a preset first throughput threshold;
before broadcasting the node service message to the corresponding node, the method further includes:
adding a first preset number of service nodes into the first set of nodes;
the broadcasting the node service message to the corresponding node comprises:
broadcasting the identification of the added serving node and the belonging information of the first set of nodes to other sets of nodes in the cluster.
5. The method of claim 1, wherein the node service message is generated when a task throughput of a second set of nodes in the cluster is less than or equal to a preset second throughput threshold;
deleting a second preset number of service nodes from the second node set;
broadcasting the identity of the deleted service node and the belonging information of the second set of nodes to other sets of nodes in the cluster.
6. The method of claim 1, wherein the node service message is generated when an operational status of a corresponding node of a third set of nodes in the cluster is abnormal;
changing a service node in the third node set into a new communication node;
and broadcasting the identification of the new communication node and the belonging information of the third node set to other node sets in the cluster.
7. The method of claim 4, further comprising:
and acquiring and recording the node states of the added service nodes in the first node set.
8. The method of claim 6, further comprising:
and acquiring and recording the node state of the new communication node in the third node set.
9. The method of claim 1, wherein the attribute information comprises location information;
the dividing the service nodes in the cluster according to the attribute information to obtain at least two node sets includes:
selecting at least two service nodes from the cluster as target nodes;
according to the position information respectively corresponding to the service nodes, when the distance between the target node and other service nodes is determined to be smaller than a preset distance threshold value, the target node and other service nodes are divided into a node set;
if the fourth service node is not divided, dividing the fourth service node into a node set closest to the fourth service node;
and carrying out node duplicate removal operation on all the node sets obtained by division to obtain at least two node sets, wherein the node duplicate removal operation comprises deleting the service nodes repeatedly divided in the node sets, and the service nodes contained in each of the at least two node sets are different.
10. An apparatus for processing a cluster message, the apparatus comprising:
the acquisition module is used for acquiring attribute information respectively corresponding to service nodes in a cluster of the data structure server;
the dividing module is used for dividing the service nodes in the cluster according to the attribute information to obtain at least two node sets;
a determining module, configured to determine communication nodes corresponding to the at least two node sets, where a communication node corresponding to each node set is a service node in the node set, the communication node corresponding to each node set is in communication connection with communication nodes in other node sets, and each service node in each node set is connected to each other;
and the broadcasting module is used for broadcasting the node service message to the communication node if the node service message generated in the cluster is monitored, so that the communication node broadcasts the node service message to other service nodes in a node set where the communication node is located, and the node service message is used for indicating at least one of node addition, node deletion and communication node change of the service node.
11. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of processing a group message as recited in any one of claims 1-9.
12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method for handling a group message according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110831096.6A CN113596746B (en) | 2021-07-22 | 2021-07-22 | Cluster message processing method and device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110831096.6A CN113596746B (en) | 2021-07-22 | 2021-07-22 | Cluster message processing method and device, electronic equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113596746A CN113596746A (en) | 2021-11-02 |
| CN113596746B true CN113596746B (en) | 2022-11-04 |
Family
ID=78249021
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110831096.6A Active CN113596746B (en) | 2021-07-22 | 2021-07-22 | Cluster message processing method and device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113596746B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114978992B (en) * | 2022-05-30 | 2023-09-12 | 中国联合网络通信集团有限公司 | Communication method, node and network of safety naming data network |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103441918A (en) * | 2013-08-29 | 2013-12-11 | 哈尔滨工程大学 | Self-organizing cluster server system and self-organizing method thereof |
| CN106790624A (en) * | 2016-12-30 | 2017-05-31 | Tcl集团股份有限公司 | New node adds the method and device of server cluster |
| CN107295080A (en) * | 2017-06-19 | 2017-10-24 | 北京百度网讯科技有限公司 | Date storage method and server applied to distributed server cluster |
| CN109792406A (en) * | 2018-07-27 | 2019-05-21 | 袁振南 | Message delivery method, device and storage medium in server cluster |
| CN112737800A (en) * | 2019-10-28 | 2021-04-30 | 顺丰科技有限公司 | Service node fault positioning method, call chain generation method and server |
| CN112799789A (en) * | 2021-03-22 | 2021-05-14 | 腾讯科技（深圳）有限公司 | Node cluster management method, device, equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7904551B2 (en) * | 2007-03-29 | 2011-03-08 | Oracle International Corporation | Unicast clustering messaging |
| CN111181801B (en) * | 2019-12-04 | 2021-11-09 | 腾讯云计算（北京）有限责任公司 | Node cluster testing method and device, electronic equipment and storage medium |
| CN112954609B (en) * | 2021-02-09 | 2023-09-05 | 北京交通大学 | Distributed geographic position service method based on backbone ring |
-
2021
- 2021-07-22 CN CN202110831096.6A patent/CN113596746B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103441918A (en) * | 2013-08-29 | 2013-12-11 | 哈尔滨工程大学 | Self-organizing cluster server system and self-organizing method thereof |
| CN106790624A (en) * | 2016-12-30 | 2017-05-31 | Tcl集团股份有限公司 | New node adds the method and device of server cluster |
| CN107295080A (en) * | 2017-06-19 | 2017-10-24 | 北京百度网讯科技有限公司 | Date storage method and server applied to distributed server cluster |
| CN109792406A (en) * | 2018-07-27 | 2019-05-21 | 袁振南 | Message delivery method, device and storage medium in server cluster |
| CN112737800A (en) * | 2019-10-28 | 2021-04-30 | 顺丰科技有限公司 | Service node fault positioning method, call chain generation method and server |
| CN112799789A (en) * | 2021-03-22 | 2021-05-14 | 腾讯科技（深圳）有限公司 | Node cluster management method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113596746A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111818112B (en) | Kafka system-based message sending method and device | |
| CN110096336B (en) | Data monitoring method, device, equipment and medium | |
| US9680692B2 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN112445626B (en) | Data processing method and device based on message middleware | |
| CN109167819B (en) | Data synchronization system, method, device and storage medium | |
| CN110858162B (en) | Memory management method and device and server | |
| CN108595483B (en) | Data processing method and related device | |
| CN110955704A (en) | Data management method, device, equipment and storage medium | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN113596746B (en) | Cluster message processing method and device, electronic equipment and medium | |
| CN108228432A (en) | A kind of distributed link tracking, analysis method and server, global scheduler | |
| CN102769495B (en) | A kind of optical fiber access network equipment communication means, Apparatus and system | |
| CN116980241A (en) | Data processing method, system, device, electronic equipment and storage medium | |
| CN108829735B (en) | Synchronization method, device, server and storage medium for parallel execution plan | |
| CN111046004A (en) | Data file storage method, device, equipment and storage medium | |
| CN113905252A (en) | Data storage method and device for live broadcast room, electronic equipment and storage medium | |
| JP2007221522A (en) | Polling device, terminal device, polling method and program | |
| WO2019214685A1 (en) | Message processing method, apparatus, and system | |
| CN114124541B (en) | Data processing method, device and equipment for rule ID and storage medium | |
| CN113132233B (en) | Data processing method, software defined network controller and data processing system | |
| CN114385081B (en) | Disk protection method for kafka cluster and related equipment | |
| CN115481187B (en) | Data reading and writing method, device and storage medium | |
| CN115983201A (en) | Global unique identifier generation method and device, electronic equipment and storage medium | |
| KR102128389B1 (en) | Cloud-based apparatus and mehtod for processing data, and cloud-based user device for receiving data process service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |