CN110601903B - Data processing method and device based on message queue middleware - Google Patents
Data processing method and device based on message queue middleware Download PDFInfo
- Publication number
- CN110601903B CN110601903B CN201910911585.5A CN201910911585A CN110601903B CN 110601903 B CN110601903 B CN 110601903B CN 201910911585 A CN201910911585 A CN 201910911585A CN 110601903 B CN110601903 B CN 110601903B
- Authority
- CN
- China
- Prior art keywords
- message server
- slave
- master
- message
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/2871—Implementation details of single intermediate entities
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to a data processing method and a device based on message queue middleware, wherein the message queue middleware comprises a management node, a master message server, a slave message server and a synchronization node, wherein the master message server and the slave message server are respectively provided with a corresponding synchronization node, and the synchronization node is used for recording summary information of messages stored in the corresponding master message server or slave message server; the method is applied to a management node and comprises the following steps: judging whether the main message server fails; and if the master message server is judged to be in fault, acquiring summary information recorded by the synchronous node from the synchronous node corresponding to the slave message server, and selecting a new master message server from the slave message servers according to the summary information. The embodiment ensures that the failure recovery of the Broker cluster is completed within a few seconds without affecting the performance of the message queue, and maximally ensures the integrity of the data.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus based on message queue middleware.
Background
Distributed Message Queue (MQ) middleware has gradually become a core means for internal communication of enterprise information systems. It has a series of functions such as low coupling, reliable delivery, broadcast, flow control, final consistency, etc., and becomes one of the main means of asynchronous RPC (Remote Procedure Call). As one of core components of a high-concurrency system, the method can help the business system to deconstruct and improve development efficiency and system stability.
The composition of MQ mainly comprises the following parts:
and (4) Broker: and the message server is used as a server to provide a message core service.
The Producer, the message Producer, the service initiator, is responsible for producing messages for transmission to the Broker.
Consumer: and the message consumer and the service processing party are responsible for acquiring the message from the Broker and performing service logic processing.
Topic, the messages under the publish-subscribe mode (Pub/Sub) are uniformly collected, different producers send messages to the Topic, and the messages are distributed to different subscribers by the MQ server, so that the broadcast of the messages is realized.
The Queue: in a queue, in a PTP (Point To Point) mode, a specific producer sends a message To a specific queue, and a consumer subscribes To the specific queue To complete reception of a specified message.
Message: and the message body is used for encapsulating service data according to the data packet which is encoded in the fixed format defined by different communication protocols to realize the transmission of the message.
However, the distributed message queue middleware disclosed in the related art does not provide an election function, when a main Broker goes down, there is a possibility that messages are lost, and the Broker cluster does not receive the writing of new messages any more, which may affect the use of the service party. After the message queue process is abnormally suspended, how to quickly ensure the election of the message queue node and the synchronization of data is a problem worthy of discussion.
Disclosure of Invention
In view of the above, the present invention is proposed to provide a data processing method and apparatus based on message queue middleware, which overcomes or at least partially solves the above problems.
In a first aspect, the present application provides a data processing method based on a message queue middleware, where the message queue middleware includes a management node, a master message server, a slave message server, and a synchronization node, where the master message server and the slave message server each have a corresponding synchronization node, and the synchronization node is configured to record summary information of messages stored in the corresponding master message server or slave message server;
the method is applied to the management node and comprises the following steps:
judging whether the main message server fails or not;
and if the master message server is judged to be in fault, acquiring summary information recorded by the synchronous node from the synchronous node corresponding to the slave message server, and selecting a new master message server from the slave message servers according to the summary information, wherein the new master message server is used for synchronizing the unsynchronized messages from the synchronous node corresponding to the fault master message server and providing services for the outside.
Optionally, the determining whether the primary message server fails includes:
if the heartbeat packet sent by the master message server is not received within a preset time length, sending a connection inquiry request to the slave message server;
if a disconnection message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server fails, wherein the disconnection message is a message generated when the slave message server determines that the connection with the master message server is disconnected;
and if a connection success message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server has no fault, wherein the connection success message is a message generated when the slave message server determines that the connection with the master message server is not disconnected.
Optionally, the electing a new master message server from the slave message servers according to the summary information includes:
calculating the difference value of the summary information recorded by the synchronous node of each slave message server and the summary information recorded by the synchronous node of the master message server, wherein the summary information comprises the message quantity of each theme and each partition, or the total size of each message file of the stored message;
and determining the slave message server corresponding to the synchronization node where the summary information with the minimum difference is located as a new master message server.
Optionally, before the determining whether the primary message server fails, the method further includes:
receiving a registration request sent by the master message server or the slave message server, wherein the registration request comprises equipment information of the master message server or the slave message server and code information indicating that the equipment information is in a master message server or slave message server role;
and according to the registration request, performing registration processing on the master message server or the slave message server, wherein the registration processing comprises the step of storing the equipment information and the corresponding code information in an associated manner.
Optionally, after the electing a new master message server from the slave message servers according to the summary information, the method further includes:
and sending the connection address of the new main message server to the rest of the slave message servers, wherein the rest of the slave message servers are used for connecting the new main message server according to the connection address.
In a second aspect, the present application provides a data processing apparatus based on a message queue middleware, where the message queue middleware includes a management node, a master message server, a slave message server, and a synchronization node, where the master message server and the slave message server each have a corresponding synchronization node, and the synchronization node is configured to record summary information of messages stored in the corresponding master message server or slave message server;
the device is applied to the management node and comprises the following steps:
the fault judging module is used for judging whether the main message server fails or not;
and the election module is used for acquiring summary information recorded by the synchronous node from the synchronous node corresponding to the slave message server if the master message server is judged to be in fault, and electing a new master message server from the slave message servers according to the summary information, wherein the new master message server is used for synchronizing the unsynchronized messages from the synchronous node corresponding to the faulty master message server and providing services for the outside.
Optionally, the fault determining module includes:
the heartbeat judgment submodule is used for sending a connection inquiry request to the slave message server if the heartbeat packet sent by the master message server is not received within a preset time length;
a connection query sub-module, configured to determine that the master message server fails if a disconnection message returned by the slave message server in response to the connection query request is received, where the disconnection message is a message generated when the slave message server determines that the connection with the master message server is disconnected; and if a connection success message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server has no fault, wherein the connection success message is a message generated when the slave message server determines that the connection with the master message server is not disconnected.
Optionally, the election module is specifically configured to:
calculating the difference value of the summary information recorded by the synchronous node of each slave message server and the summary information recorded by the synchronous node of the master message server, wherein the summary information comprises the message quantity of each theme and each partition, or the total size of each message file of the stored message;
and determining the slave message server corresponding to the synchronization node where the summary information with the minimum difference is located as a new master message server.
Optionally, the apparatus further comprises:
a registration module, configured to receive a registration request sent by the master message server or the slave message server before determining whether the master message server fails, where the registration request includes device information of the master message server or the slave message server and code information indicating that the registration request is a role of the master message server or the slave message server; and according to the registration request, performing registration processing on the master message server or the slave message server, wherein the registration processing comprises the step of storing the equipment information and the corresponding code information in an associated manner.
Optionally, the apparatus further comprises:
and a connection address sending module, configured to send a connection address of the new master message server to remaining slave message servers after the new master message server is elected from the slave message servers according to the summary information, where the remaining slave message servers are used to connect to the new master message server according to the connection address.
In a third aspect, the present application provides an electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method as described above.
In a fourth aspect, the present application provides a storage medium having instructions that, when executed by a processor of the device, enable the electronic device to perform the method as described above.
The invention has the following beneficial effects:
in this embodiment, when it is determined that the Master message server Broker Master fails, election of a new Master message server and synchronization of data are both performed by the management node Mq Manager, the original Mq only needs to be capable of receiving a command, and switching from the Broker Slave to the Broker Master is performed, and processing of the rest of the parts, such as synchronization of data, is a self-contained function of the original Mq.
In addition, the election of a new main message server is realized through an independent management node, the synchronization of messages and the recording of the summary information of the messages are carried out through the synchronization node, the fault recovery of the Broker cluster can be guaranteed to be completed within a few seconds on the premise that the performance of a message queue is not affected, and the integrity of data is guaranteed in the best effort.
Drawings
FIG. 1 is a diagram illustrating an architecture of message queue middleware according to the present application;
FIG. 2 is a flowchart illustrating steps of an embodiment of a data processing method based on message queue middleware according to the present application;
fig. 3 is a block diagram of an embodiment of a data processing apparatus based on message queue middleware according to the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Referring to fig. 1, a schematic diagram of an architecture of a message queue middleware of the present application is shown, where the message queue middleware may include at least a management node Mq Manager, a Master message server Broker Master, a Slave message server Broker Slave, and a synchronization node Mq SyncNode.
The number of the breaker masters can be one or more, the number of the breaker Slave can be multiple, and the breaker masters and the breaker Slave form a breaker cluster. In the Broker cluster, the Broker Master mainly carries read-write requests, and the Broker Slave carries partial read requests, while synchronizing data from the Broker Master.
In this embodiment, the management node Mq Manager is disposed in the message queue middleware to manage the message queue process, for example, including election, switching, heartbeat detection, and the like of the Broker Master.
In one example, a Broker of a message queue can obtain management of the Mq Manager by registering with the Mq Manager. The Mq Manager realizes the registration of the Broker Master or the Broker Slave by receiving a registration request sent by the Broker Master or the Broker Slave.
Illustratively, the registration request may include device information of the Broker Master or the Broker Slave, and code information indicating that it is a role of the Broker Master or the Broker Slave. The device information may include a device ID, a connection address (e.g., an IP address), and the like. The code information may be a preset code representing each character, for example, if the Broker is the Broker Master, the code information may be a value of 0; if the breaker is a breaker Slave, the code information may be a value of 1.
After receiving the registration request, the Mq Manager can store the device information of the Broker and the corresponding code information in an associated manner, so that the Mq Manager can know which Broker cluster is the Broker Master and which Broker Slave is the Broker Slave, and subsequent judgment and election of the fault of the Broker Master are facilitated.
After the registration is completed, the Broker can send a heartbeat message to the Mq Manager at regular time, so that the Mq Manager can perform heartbeat detection on the Broker to confirm whether the Broker fails. Meanwhile, the MQ SyncNode corresponding to the Broker starts to record data on the Broker.
It should be noted that, in order to prevent the election process of the present solution from being unavailable due to a failure of the Mq Manager and facilitate management of the Mq Manager, in this embodiment, a plurality of Mq Manager instances may be deployed, and a unique serial number is configured for each Mq Manager instance (only one Mq Manager instance is needed to be responsible for election), and when the current Mq Manager fails, the Mq Manager of the next serial number may be used to continue to provide services. In other embodiments, an Mq Manager may also be elected from the Mq sync nodes, for example, a unique serial number is configured for each Mq sync node, and when the current Mq Manager fails, the Mq sync node with the next serial number may be used as a new Mq Manager to continue providing services. In addition, in this embodiment, a synchronization node Mq syncNode is deployed for each message queue process, that is, each Broker (including the Broker Master and the Broker Slave), and is used to synchronize messages in the corresponding Broker Master or the Broker Slave, and record summary information of messages stored in the corresponding Broker Master or the Broker Slave, and the like.
As an example, the summary information may include the number of messages for each topic and each partition in the Broker. For example, Mq syncNode may record summary information as follows:
assuming topic a, 10 messages for part 1 and 20 messages for part 2, the summary information recorded in Mq syncNode is as follows:
as another example, the summary information may include the total size of each message file in the Broker that holds the messages (since messages are all written to, and not modified, incremental synchronization may be performed based on file size). For example, Mq syncNode may record summary information as follows:
assuming that msg _ a files in the message queue hold 20MB of messages and msg _ b holds 30MB of messages, Mq syncNode can record as follows:
based on the message queue middleware of fig. 1, referring to fig. 2, a flowchart of steps of an embodiment of a data processing method based on the message queue middleware of the present application is shown. When the present embodiment is applied to a management node, the following steps may be specifically included:
In this embodiment, the Mq Manager may determine whether a breaker Master in the breaker cluster has a fault such as downtime.
In one embodiment, the Mq Manager may determine whether the Broker Master fails by:
if the heartbeat packet sent by the master message server is not received within a preset time length, sending a connection inquiry request to the slave message server; if a disconnection message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server fails, wherein the disconnection message is a message generated when the slave message server determines that the connection with the master message server is disconnected; and if a connection success message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server has no fault, wherein the connection success message is a message generated when the slave message server determines that the connection with the master message server is not disconnected.
Specifically, in a distributed scenario, communication between the Mq Manager and the Broker Master may be delayed or communication network is abnormal, so that the Mq Manager may erroneously determine that the Broker Master fails. To avoid this, in this embodiment, when the Mq Manager does not receive the heartbeat packet sent by the Broker Master within the preset time period, that is, the Mq Manager is disconnected from the heartbeat of the Broker Master, the Mq Manager may send a connection inquiry request to a part or all of the Broker Slave corresponding to the Broker Master to inquire whether the Broker Slave is connectable with the Broker Master, and if the Broker Slave confirms that the Broker Slave is not connectable with the Broker Master, a connection disconnection message is sent to the Mq Manager, and the Mq Manager may determine that the Broker Master fails according to the connection disconnection message. Otherwise, if the Broker Slave confirms that the connection with the Broker Master is successful, a connection success message is sent to the Mq Manager, and the Mq Manager can determine that the Broker Master does not have a fault according to the connection success message.
And 102, if the master message server is judged to be in fault, acquiring summary information recorded by the synchronous node from the synchronous node corresponding to the slave message server, and selecting a new master message server from the slave message servers according to the summary information.
In the step, when the Mq Manager determines that the Broker Master of the current Broker cluster is in fault, a new Broker Master election is performed, and because the election is controlled by an Mq Manager single node, a concurrent election problem does not exist (the concurrent election may cause multiple election results to cause abnormality), so that the new Broker Master election can be guaranteed to be completed within second-level time.
In this embodiment, the Mq Manager may obtain summary information recorded by the Mq sync node corresponding to each Broker in the Broker cluster, and when determining a machine fault corresponding to the Broker Master, may select a new Broker Master from the plurality of Broker slaves according to the summary information recorded by the Mq sync node of each Broker Slave corresponding to the Broker Master.
In an embodiment, the Mq Manager may calculate a difference between summary information recorded by an Mq syncnnode of each Broker Slave and summary information recorded by an Mq syncnnode of the Broker Master, and determine the Broker Slave corresponding to the Mq syncnnode where the summary information with the smallest difference is located as the new Broker Master.
In implementation, the calculation method of the difference may be different according to the recording method of the Mq syncNode. For example, for the case that the summary information recorded by the Mq syncNode is the number of messages of each topic and each partition, it is assumed that the number of messages of topic1 recorded by the Mq syncNode of the Broker Master is 30, the number of messages of topic1 recorded by the Mq syncNode corresponding to the Broker Slave 1-5 is 20, 23, 18, 25, and 21, and the corresponding difference values are: 10, 7, 12, 5 and 9, the breaker Slave 4 can be used as a new breaker Master. For another example, for the case that the summary information recorded by the Mq syncNode is the total size of each message file for storing messages, assuming that the total size of the message file 1 recorded by the Mq syncNode of the Broker Master is 20MB, the total size of the message file 1 recorded by the Mq syncNode corresponding to the Broker Slave 1-5 is 20MB, 18MB, 19MB, 15MB, and 10MB, respectively, and the corresponding difference values are: 0MB, 2MB, 1MB, 5MB and 10MB, the Broker Slave 1 can be used as a new Broker Master.
Of course, the embodiment is not limited to the above two ways of recording messages and choosing a new Broker Master, and other choosing ways of determining the Mq Manager by those skilled in the art according to actual requirements are all possible.
After selecting a new Broker Master, the new Broker Master can synchronize the unsynchronized messages from the Mq sync node corresponding to the failed Broker Master and provide services to the outside. In one embodiment, the Mq Manager may send the connection address of the new Broker Master to the remaining Broker Slave, which is used to connect the new Broker Master according to the connection address.
In practice, if the Mq SyncNode of the original Broker Master can still connect, the Mq SyncNode node of the new Broker Master can synchronize the messages that have not been synchronized from the Mq SyncNode of the original Broker Master to ensure that the new Broker Master has all the messages, and such synchronization can be completed in seconds. In other embodiments, if the Mq SyncNode of the original Broker Master cannot be connected, the new Broker Master may directly provide external services without performing message synchronization. For the latter embodiment, a small number of messages may be lost, and this situation may be avoided by a semi-synchronous copy manner provided by the MQ, that is, after the messages are sent to the Broker Master, the Broker Master needs to wait for the Broker Master to synchronize the messages to one or more of the external Broker Slave before the messages are successfully sent.
After the synchronization of the new Broker Master is completed, the Mq Manager may send a connection address of the new Broker Master to the remaining Broker Slave, and the remaining Broker Slave in the current Broker cluster may perform data synchronization directly with the new Broker Master through the connection address, so that the current Broker cluster may provide a service to the outside again within several seconds.
In this embodiment, when it is determined that the Master message server Broker Master fails, election of a new Master message server and synchronization of data are both performed by the management node Mq Manager, the original Mq only needs to be capable of receiving a command, and switching from the Broker Slave to the Broker Master is performed, and processing of the rest of the parts, such as synchronization of data, is a self-contained function of the original Mq.
In addition, the election of a new main message server is realized through an independent management node, the synchronization of messages and the recording of the summary information of the messages are carried out through the synchronization node, the fault recovery of the Broker cluster can be guaranteed to be completed within a few seconds on the premise that the performance of a message queue is not affected, and the integrity of data is guaranteed in the best effort.
Based on the above data processing method based on the message queue middleware, referring to fig. 3, a structural block diagram of an embodiment of the data processing device based on the message queue middleware of the present invention is shown, where the message queue middleware includes a management node, a master message server, a slave message server, and a synchronization node, where the master message server and the slave message server each have a corresponding synchronization node, and the synchronization node is used to record summary information of messages stored in the corresponding master message server or slave message server;
the device is applied to the management node and can comprise the following modules:
a failure determining module 301, configured to determine whether the primary message server fails;
an election module 302, configured to, if it is determined that the master message server fails, obtain summary information recorded by the synchronization node from the synchronization node corresponding to the slave message server, and elect a new master message server from the slave message servers according to the summary information, where the new master message server is used to synchronize an unsynchronized message from the synchronization node corresponding to the failed master message server and provide a service to the outside.
In one embodiment, the failure determination module 301 includes:
the heartbeat judgment submodule is used for sending a connection inquiry request to the slave message server if the heartbeat packet sent by the master message server is not received within a preset time length;
a connection query sub-module, configured to determine that the master message server fails if a disconnection message returned by the slave message server in response to the connection query request is received, where the disconnection message is a message generated when the slave message server determines that the connection with the master message server is disconnected; and if a connection success message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server has no fault, wherein the connection success message is a message generated when the slave message server determines that the connection with the master message server is not disconnected.
In an embodiment, the election module 302 is specifically configured to:
calculating the difference value of the summary information recorded by the synchronous node of each slave message server and the summary information recorded by the synchronous node of the master message server, wherein the summary information comprises the message quantity of each theme and each partition, or the total size of each message file of the stored message;
and determining the slave message server corresponding to the synchronization node where the summary information with the minimum difference is located as a new master message server.
In one embodiment, the apparatus further comprises:
a registration module, configured to receive a registration request sent by the master message server or the slave message server before determining whether the master message server fails, where the registration request includes device information of the master message server or the slave message server and code information indicating that the registration request is a role of the master message server or the slave message server; and according to the registration request, performing registration processing on the master message server or the slave message server, wherein the registration processing comprises the step of storing the equipment information and the corresponding code information in an associated manner.
In one embodiment, the apparatus further comprises:
and a connection address sending module, configured to send a connection address of the new master message server to remaining slave message servers after the new master message server is elected from the slave message servers according to the summary information, where the remaining slave message servers are used to connect to the new master message server according to the connection address.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Further, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (12)
1. The data processing method based on the message queue middleware is characterized in that the message queue middleware comprises a management node, a master message server, a slave message server and a synchronization node, wherein the master message server and the slave message server are respectively provided with a corresponding synchronization node, and the synchronization node is used for synchronizing messages in the corresponding master message server or slave message server and recording summary information of the stored messages in the corresponding master message server or slave message server;
the method is applied to the management node and comprises the following steps:
judging whether the main message server fails or not;
and if the master message server is judged to be in fault, acquiring summary information recorded by the synchronous node from the synchronous node corresponding to the slave message server, and selecting a new master message server from the slave message servers according to the summary information, wherein the new master message server is used for synchronizing the unsynchronized messages from the synchronous node corresponding to the fault master message server and providing services for the outside.
2. The method of claim 1, wherein said determining whether said primary message server is down comprises:
if the heartbeat packet sent by the master message server is not received within a preset time length, sending a connection inquiry request to the slave message server;
if a disconnection message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server fails, wherein the disconnection message is a message generated when the slave message server determines that the connection with the master message server is disconnected;
and if a connection success message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server has no fault, wherein the connection success message is a message generated when the slave message server determines that the connection with the master message server is not disconnected.
3. The method of claim 1 or 2, wherein the electing a new master message server from the slave message servers according to the aggregated information comprises:
calculating the difference value of the summary information recorded by the synchronous node of each slave message server and the summary information recorded by the synchronous node of the master message server, wherein the summary information comprises the message quantity of each theme and each partition, or the total size of each message file of the stored message;
and determining the slave message server corresponding to the synchronization node where the summary information with the minimum difference is located as a new master message server.
4. The method of claim 1, wherein prior to said determining whether said primary message server is down, said method further comprises:
receiving a registration request sent by the master message server or the slave message server, wherein the registration request comprises equipment information of the master message server or the slave message server and code information indicating that the equipment information is in a master message server or slave message server role;
and according to the registration request, performing registration processing on the master message server or the slave message server, wherein the registration processing comprises the step of storing the equipment information and the corresponding code information in an associated manner.
5. The method of claim 1, 2 or 4, wherein after said electing a new master message server from said slave message servers based on said aggregated information, the method further comprises:
and sending the connection address of the new main message server to the rest of the slave message servers, wherein the rest of the slave message servers are used for connecting the new main message server according to the connection address.
6. The data processing device is characterized in that the message queue middleware comprises a management node, a master message server, a slave message server and a synchronization node, wherein the master message server and the slave message server are respectively provided with a corresponding synchronization node, and the synchronization node is used for synchronizing messages in the corresponding master message server or slave message server and recording summary information of the stored messages in the corresponding master message server or slave message server;
the device is applied to the management node and comprises the following steps:
the fault judging module is used for judging whether the main message server fails or not;
and the election module is used for acquiring summary information recorded by the synchronous node from the synchronous node corresponding to the slave message server if the master message server is judged to be in fault, and electing a new master message server from the slave message servers according to the summary information, wherein the new master message server is used for synchronizing the unsynchronized messages from the synchronous node corresponding to the faulty master message server and providing services for the outside.
7. The apparatus of claim 6, wherein the failure determination module comprises:
the heartbeat judgment submodule is used for sending a connection inquiry request to the slave message server if the heartbeat packet sent by the master message server is not received within a preset time length;
a connection query sub-module, configured to determine that the master message server fails if a disconnection message returned by the slave message server in response to the connection query request is received, where the disconnection message is a message generated when the slave message server determines that the connection with the master message server is disconnected; and if a connection success message returned by the slave message server aiming at the connection inquiry request is received, judging that the master message server has no fault, wherein the connection success message is a message generated when the slave message server determines that the connection with the master message server is not disconnected.
8. The apparatus according to claim 6 or 7, wherein the election module is specifically configured to:
calculating the difference value of the summary information recorded by the synchronous node of each slave message server and the summary information recorded by the synchronous node of the master message server, wherein the summary information comprises the message quantity of each theme and each partition, or the total size of each message file of the stored message;
and determining the slave message server corresponding to the synchronization node where the summary information with the minimum difference is located as a new master message server.
9. The apparatus of claim 6, further comprising:
a registration module, configured to receive a registration request sent by the master message server or the slave message server before determining whether the master message server fails, where the registration request includes device information of the master message server or the slave message server and code information indicating that the registration request is a role of the master message server or the slave message server; and according to the registration request, performing registration processing on the master message server or the slave message server, wherein the registration processing comprises the step of storing the equipment information and the corresponding code information in an associated manner.
10. The apparatus of claim 6, 7 or 9, further comprising:
and a connection address sending module, configured to send a connection address of the new master message server to remaining slave message servers after the new master message server is elected from the slave message servers according to the summary information, where the remaining slave message servers are used to connect to the new master message server according to the connection address.
11. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method of any one of claims 1-5.
12. A storage medium having instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910911585.5A CN110601903B (en) | 2019-09-25 | 2019-09-25 | Data processing method and device based on message queue middleware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910911585.5A CN110601903B (en) | 2019-09-25 | 2019-09-25 | Data processing method and device based on message queue middleware |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110601903A CN110601903A (en) | 2019-12-20 |
| CN110601903B true CN110601903B (en) | 2022-04-01 |
Family
ID=68863333
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910911585.5A Active CN110601903B (en) | 2019-09-25 | 2019-09-25 | Data processing method and device based on message queue middleware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110601903B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112527520A (en) * | 2020-12-01 | 2021-03-19 | 中国建设银行股份有限公司 | Method and device for deploying message middleware |
| CN112822238B (en) * | 2020-12-29 | 2023-05-26 | 深圳市金证科技股份有限公司 | Main node switching method and computer readable storage medium |
| CN112804332B (en) * | 2021-01-14 | 2023-02-28 | 广州虎牙科技有限公司 | Message processing system, method, device, equipment and computer readable storage medium |
| CN112564990B (en) * | 2021-02-23 | 2021-05-14 | 全时云商务服务股份有限公司 | Management method for switching audio management server |
| CN113641511B (en) * | 2021-07-09 | 2024-06-04 | 阿里云计算有限公司 | Message communication method and device |
| CN114116258B (en) * | 2021-11-12 | 2024-05-07 | 苏州浪潮智能科技有限公司 | Queue manager hot standby method, system, terminal and storage medium |
| CN114598593B (en) * | 2022-02-16 | 2023-08-29 | 阿里巴巴（中国）有限公司 | Message processing method, system, computing device and computer storage medium |
| CN115086153B (en) * | 2022-05-20 | 2024-05-28 | 阿里巴巴（中国）有限公司 | Message processing system, message processing method, device and storage medium |
| CN116893914A (en) * | 2023-09-11 | 2023-10-17 | 中移(苏州)软件技术有限公司 | Message processing method, message queue system, client and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN109669820A (en) * | 2018-12-24 | 2019-04-23 | 广州君海网络科技有限公司 | Task monitoring and managing method and device based on Kettle |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8577976B2 (en) * | 2010-04-27 | 2013-11-05 | International Business Machines Corporation | Application of system level policy in message oriented middleware |
-
2019
- 2019-09-25 CN CN201910911585.5A patent/CN110601903B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106878473A (en) * | 2017-04-20 | 2017-06-20 | 腾讯科技（深圳）有限公司 | A kind of message treatment method, server cluster and system |
| CN109669820A (en) * | 2018-12-24 | 2019-04-23 | 广州君海网络科技有限公司 | Task monitoring and managing method and device based on Kettle |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110601903A (en) | 2019-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110601903B (en) | Data processing method and device based on message queue middleware | |
| US10764369B2 (en) | Data storage method and server applicable to distributed server cluster | |
| CN107465767B (en) | Data synchronization method and system | |
| CN111368002A (en) | Data processing method, system, computer equipment and storage medium | |
| CN113641511B (en) | Message communication method and device | |
| CN102143194A (en) | Data synchronization method and system, immediate data node and terminal data node | |
| CN112804332B (en) | Message processing system, method, device, equipment and computer readable storage medium | |
| CN110677282B (en) | Hot backup method of distributed system and distributed system | |
| CN107682169B (en) | Method and device for sending message by Kafka cluster | |
| CN112052230B (en) | Multi-machine room data synchronization method, computing device and storage medium | |
| CN110069365B (en) | Method for managing database and corresponding device, computer readable storage medium | |
| WO2012171346A1 (en) | Telephone number mapping-domain name system (enum-dns) and disaster tolerance method thereof | |
| CN108509296B (en) | Method and system for processing equipment fault | |
| CN114625566A (en) | Data disaster tolerance method and device, electronic equipment and storage medium | |
| CN105812492A (en) | Data synchronizing method and system | |
| CN111211927B (en) | Resource synchronization method and device | |
| CN111510336B (en) | Network equipment state management method and device | |
| CN114422335A (en) | Communication method, communication device, server and storage medium | |
| CN114490188A (en) | Method and device for synchronizing main database and standby database | |
| CN114598593A (en) | Message processing method, system, computing device and computer storage medium | |
| CN113890817A (en) | Communication optimization method and device | |
| CN111083182B (en) | Distributed Internet of things equipment management method and device | |
| CN113890880A (en) | Method, system, equipment and storage medium for data synchronization among multiple nodes | |
| CN110716827A (en) | Hot backup method suitable for distributed system and distributed system | |
| CN110890989A (en) | Channel connection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |