CN114513513A - Data processing method and device based on message middleware - Google Patents

Data processing method and device based on message middleware Download PDF

Info

Publication number
CN114513513A
CN114513513A CN202210138474.7A CN202210138474A CN114513513A CN 114513513 A CN114513513 A CN 114513513A CN 202210138474 A CN202210138474 A CN 202210138474A CN 114513513 A CN114513513 A CN 114513513A
Authority
CN
China
Prior art keywords
cluster
message
duplicator
producer
replicator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210138474.7A
Other languages
Chinese (zh)
Inventor
喻俊
张志广
刘慧中
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hunan Happly Sunshine Interactive Entertainment Media Co Ltd
Original Assignee
Hunan Happly Sunshine Interactive Entertainment Media Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hunan Happly Sunshine Interactive Entertainment Media Co Ltd filed Critical Hunan Happly Sunshine Interactive Entertainment Media Co Ltd
Priority to CN202210138474.7A priority Critical patent/CN114513513A/en
Publication of CN114513513A publication Critical patent/CN114513513A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/543—User-generated data transfer, e.g. clipboards, dynamic data exchange [DDE], object linking and embedding [OLE]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application discloses a data processing method and a device based on message middleware, wherein the method comprises the following steps: storing the message produced by the producer of the first cluster into the target theme of the first cluster; based on the consumption cursor of the first duplicator, obtaining a message from a message queue corresponding to the target subject, and storing the message into the first duplicator; calling a producer in the first duplicator, and sending the message in the first duplicator to the second cluster so that the target theme of the second cluster stores the message; after determining that the target topic of the second cluster successfully saves the message, sending a confirmation character to the first replicator to cause the first replicator to update the consumption cursors. The method is based on a first duplicator which is preset in a first cluster (specifically arranged in a brooker node process) and comprises a producer of a second cluster, and can duplicate the message of the first cluster to the second cluster under the condition of not consuming the machine resource of the first cluster, thereby improving the disaster tolerance capability of the cluster.

Description

Data processing method and device based on message middleware
Technical Field
The present application relates to the field of message middleware, and in particular, to a data processing method and apparatus based on message middleware.
Background
Message middleware is often used for high-concurrency peak clipping and valley filling and serving for decoupling among services in the whole development system, and is a very core and indispensable ring. The high availability of the message middleware is the capacity of the core of the message middleware, and the reliability and the availability of the message middleware cluster (called the cluster for short) in the same area can be ensured in a multi-copy and multi-node mode. When an abnormal cluster of an individual region occurs (such as power failure, network outage and optical fiber failure), a consumer of the abnormal cluster needs to consume other clusters (namely cross-region consumption behaviors) across the region, and for the cluster bearing the cross-region consumption behaviors, messages in the abnormal cluster need to be copied into the abnormal cluster in advance.
At present, the existing cluster message replication method generally includes: messages contained by a cluster are sent to another cluster using a separate synchronization tool service. In the existing cluster message replication mode, a cluster needs additional machine resource deployment synchronization tool service, and performance loss is caused by additional cross-process and cross-network data replication, so that the performance of the cluster is seriously affected, and the disaster tolerance capability of the cluster is reduced.
Disclosure of Invention
The application provides a data processing method and device based on message middleware, which are used for copying messages in a cluster to other clusters under the condition of not influencing the performance of the cluster, so that the disaster tolerance capability of the cluster is improved.
In order to achieve the above object, the present application provides the following technical solutions:
a data processing method based on message middleware comprises the following steps:
storing a message produced by a producer of a first cluster into a target topic of the first cluster; the message comprises a data item; the target theme is a theme corresponding to the service type to which the data item belongs;
based on a consumption cursor of a first replicator, obtaining the message from a message queue corresponding to a target topic of the first cluster, and saving the message to the first replicator; the first replicator is provisioned in the first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption vernier of the first duplicator is pre-established based on the first duplicator as a consumer of the first cluster;
calling a producer in the first duplicator to send the message stored in the first duplicator to the second cluster so that a target theme of the second cluster stores the message;
after determining that the target subject of the second cluster successfully saves the message, sending a confirmation character to the first replicator to enable the first replicator to update its own consumption cursors.
Optionally, the method further includes:
and under the condition that the first cluster fails, saving the message produced by the producer of the first cluster into the target theme of the second cluster.
Optionally, the method further includes:
under the condition that the second cluster does not fail, storing the message produced by the producer of the second cluster into a target topic of the second cluster;
based on a consumption cursor of a second replicator, obtaining the message from a message queue corresponding to a target topic of the second cluster, and saving the message to the second replicator; the second replicator is provisioned in the second cluster and comprises a producer of the first cluster; the consumption vernier of the second duplicator is pre-established based on the second duplicator as a consumer of the second cluster;
calling a producer in the second duplicator to send the message stored in the second duplicator to the first cluster so that a target subject of the first cluster stores the message;
after determining that the message is successfully saved by the target subject of the first cluster, sending a confirmation character to the second replicator to enable the second replicator to update its own consumption cursors.
Optionally, the message further includes metadata of the data item, the metadata including an ID of the message;
the saving the message produced by the producer of the second cluster to the target topic of the second cluster comprises:
and in the case that the value of the ID of the message produced by the producer of the second cluster is not larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, saving the message to the target subject of the second cluster.
Optionally, the method further includes:
and if the value of the ID of the message produced by the producer of the second cluster is larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, forbidding to save the message in the second cluster and deleting the message.
Optionally, the message further includes metadata of the data item, the metadata including a cluster identifier of a cluster to which a producer of the data item belongs;
the calling a producer in the second duplicator to send the message saved in the second duplicator to the first cluster so that the target subject of the first cluster saves the message comprises:
and under the condition that the message stored in the second duplicator does not contain the cluster identification of the first cluster, calling a producer in the second duplicator, and sending the message stored in the second duplicator to the first cluster so as to enable the target subject of the first cluster to store the message.
Optionally, the method further includes:
and if the message stored in the second duplicator contains the cluster identifier of the first cluster, prohibiting the message stored in the second duplicator from being sent to the first cluster.
A message-middleware based data processing apparatus, comprising:
the message storage unit is used for storing the message produced by the producer of the first cluster into the target theme of the first cluster; the message comprises a data item; the target theme is a theme corresponding to the service type to which the data item belongs;
the message consumption unit is used for acquiring the message from a message queue corresponding to the target subject of the first cluster based on the consumption cursor of the first duplicator and storing the message into the first duplicator; the first replicator is provisioned in the first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption vernier of the first duplicator is pre-established based on the first duplicator as a consumer of the first cluster;
a message replication unit, configured to invoke a producer in the first replicator, and send the message stored in the first replicator to the second cluster, so that a target topic of the second cluster stores the message;
and the cursor updating unit is used for sending a confirmation character to the first duplicator after the target theme of the second cluster is determined to successfully save the message, so that the first duplicator updates the self consumption cursor.
A computer-readable storage medium comprising a stored program, wherein the program executes the message-middleware based data processing method.
A message middleware based data processing apparatus comprising: a processor, a memory, and a bus; the processor and the memory are connected through the bus;
the memory is used for storing a program, and the processor is used for executing the program, wherein the program executes the data processing method based on the message middleware during running.
According to the technical scheme, the message produced by the producer of the first cluster is stored in the target theme of the first cluster. Based on the consumption cursor of the first replicator, messages are retrieved from a message queue corresponding to the target topic of the first cluster and saved to the first replicator. And calling a producer in the first duplicator to send the message stored in the first duplicator to the second cluster so that the target theme of the second cluster stores the message. After determining that the target topic of the second cluster successfully saves the message, sending a confirmation character to the first replicator to cause the first replicator to update its own consumption curser. Compared with the prior art, the scheme disclosed by the application is based on the first duplicator which is preset in the first cluster and contains the producer of the second cluster, and the message contained in the first cluster can be duplicated to the second cluster under the condition that the machine resource of the first cluster is not consumed, so that the message in the cluster is duplicated to other clusters under the condition that the performance of the cluster is not influenced, and the disaster tolerance capability of the cluster is effectively improved. In addition, because the first duplicator can be arranged in the browser node process of the first cluster, the cross-process and cross-network duplication loss among data can be eliminated, and the speed of duplicating the information of the first cluster to the second cluster is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1a is a schematic flowchart of a data processing method based on message middleware according to an embodiment of the present application;
fig. 1b is a schematic diagram of a data processing scenario provided in an embodiment of the present application;
fig. 2 is a schematic flowchart of another data processing method based on message middleware according to an embodiment of the present application;
fig. 3 is a schematic architecture diagram of a data processing apparatus based on message middleware according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
As shown in fig. 1a, a schematic flowchart of a data processing method based on message middleware provided in an embodiment of the present application includes the following steps:
s101: based on the first replicator as a consumer of the first cluster, a consumption cursor of the first replicator is pre-established.
The first duplicator is preset in a first cluster and comprises a producer of a second cluster, and the first cluster and the second cluster belong to different areas respectively. The consumption cursor of the first replicator is used to indicate the IDs of messages in the message queue shown by the first cluster that can be consumed by the first replicator. Generally, the first replicator may be built into the broker node process of the first cluster, so-called broker node process, which is well known to those skilled in the art and will not be described herein.
Since the first cluster is the message middleware, based on the characteristics of the message middleware, it can be known that: only if the first replicator subscribes to each topic in the first cluster, the first cluster pushes messages which can be consumed by the first replicator in a message queue (in the embodiment of the present application, the topic can be equivalent to the message queue) corresponding to each topic to the first replicator according to the consumption cursor of the first replicator.
It should be noted that the number of first replicators included in the first cluster is the same as the number of topics of the first cluster, and of course, the number of first replicators included in the first cluster is also the same as the number of topics of the second cluster. That is, the number of topics contained in the first cluster is the same as the number of topics contained in the second cluster. In the embodiment of the present application, the topic names included in the first cluster and the second cluster are also the same, for example, the first cluster includes a username topic, and the second cluster also includes a username topic.
S102: and under the condition that the first cluster does not fail, saving the message produced by the producer of the first cluster into the target topic of the first cluster.
The message comprises a data item and metadata of the data item, wherein the metadata at least comprises an ID of the message and a cluster identifier of a cluster to which a producer of the data item belongs, and specifically, the ID of the message is determined based on a name of the producer of the data item and a unique incremental ID of the data item, so that the ID of each message is ensured to be different. In the embodiment of the present application, the target topic is a topic corresponding to a service type to which the data item belongs.
Based on the characteristics of the message middleware, before a producer sends a message to the first cluster, the producer carries verification sending data Schema to the first cluster for compatibility verification and connection establishment, and after the producer establishes connection with the first cluster, the first cluster performs distributed storage (i.e. storing the message in a preset distributed storage) on the message sent by the producer and informs the producer that the message is successfully sent.
In addition, the consumer of the first cluster establishes connection with the first cluster through the verification received data Schema and sets own consumption cursor. Based on the characteristics of the message middleware, the first cluster acquires the messages which can be consumed by the consumers from the message queue corresponding to the target topic according to the consumption cursors of the consumers who subscribe to the target topic, and pushes the messages which can be consumed by the consumers to the consumers.
S103: based on the consumption cursor of the first replicator, messages are retrieved from a message queue corresponding to the target topic of the first cluster and saved to the first replicator.
S104: and calling a producer in the first duplicator to send the message stored in the first duplicator to the second cluster so that the target theme of the second cluster stores the message.
Based on the producer of the second cluster included in the first duplicator, the message stored in the first duplicator can be synchronously sent to the second cluster, and the message stored in the first duplicator can be asynchronously sent to the second cluster, so that the influence of the first duplicator on the consumption behavior of the consumer of the first cluster is effectively reduced.
S105: after determining that the target topic of the second cluster successfully saves the message, an Acknowledgement Character (ACK) is sent to the first replicator to cause the first replicator to update its own consumption cursors.
After the producer in the first duplicator successfully sends the message to the second cluster, the first duplicator needs to be informed that the message is successfully sent, so that the first duplicator is prevented from wasting time and sending the message again, and the consumption cursors of the first duplicator are updated in time so as to send other messages to the second cluster as soon as possible.
Based on the above flow shown in S101-S105, the scenario shown in fig. 1b can be briefly summarized, and the specific implementation process of the scenario shown in fig. 1b includes:
1. after a producer A sends a first message to a topic A of a first cluster, acquiring the first message from a message queue shown by the topic A of the first cluster based on a consumption cursor of a first duplicator, and storing the first message into the first duplicator;
2. when a first cluster pushes a first message to a consumer A subscribing to a subject A of the first cluster, a producer in a first duplicator is called, and the first message stored in the first duplicator is sent to the subject A of a second cluster, so that the subject A of the second cluster stores the first message;
3. after determining that the topic A of the second cluster successfully stores the first message, sending ACK (acknowledgement) to the first duplicator so that the first duplicator updates the consumption vernier of the first duplicator;
4. after the producer B sends a second message to the theme A of the second cluster, acquiring a first message and a second message from a message queue shown by the theme A of the second cluster based on a consumption cursor of a second duplicator, and storing the first message and the second message into the second duplicator;
5. and controlling the second cluster to push the first message and the second message to a consumer B subscribed to the self topic A.
It can be known from the above specific implementation process that, based on the first replicator preset in the first cluster, the first cluster replicates its own message to the second cluster without consuming machine resources, thereby ensuring that the performance of the first cluster is not affected, and for the second cluster, since the first replicator includes a producer of the second cluster, based on the data transmission characteristics between the producer and the message middleware, it can use the producer in the first replicator to replicate the message in the first cluster to the second cluster, and likewise, it cannot occupy the machine resources of the second cluster, thereby ensuring that the performance of the second cluster is not affected. In addition, because the first duplicator can be arranged in the brooker node process of the first cluster, the cross-process and cross-network duplication loss among data can be eliminated, and the speed of duplicating the information of the first cluster to the second cluster is improved.
It should be emphasized that the second cluster may also be provided with a replicator, and the first cluster and the second cluster can perform data replication with each other based on the replicator in the second cluster, and a specific implementation process may be shown in S106-110 described below.
S106: based on the second replicator as a consumer of the second cluster, a consumption cursor of the second replicator is pre-established.
Wherein the second replicator is provisioned in a second cluster and comprises a producer of the first cluster. The consumption cursor of the second replicator is used to indicate the IDs of messages in the message queue shown by the second cluster that can be consumed by the second replicator. In general, a second replicator may be built into the Broker node process of the second cluster.
Since the second cluster is message middleware, based on the characteristics of the message middleware, it is known that: only the second replicator has subscribed to each topic in the second cluster, the second cluster pushes messages which can be consumed by the second replicator in the message queue corresponding to each topic to the second replicator according to the consumption cursor of the second replicator.
S107: and in the case that the second cluster does not fail, saving the message produced by the producer of the second cluster into the target topic of the second cluster, and in the case that the first cluster fails, saving the message produced by the producer of the first cluster into the target topic of the second cluster.
And saving the message to the target subject of the second cluster under the condition that the value of the message ID of the message produced by the producer of the second cluster is not larger than the value of the message ID shown by the consumption cursor of the second duplicator.
Optionally, in a case that the value of the message ID of the message produced by the producer of the second cluster is greater than the value of the message ID indicated by the consumption cursor of the second replicator, the message is prohibited from being stored in the second cluster and deleted, thereby ensuring the idempotency of the data.
It should be noted that the message produced by the producer of the second cluster and the message produced by the producer of the first cluster each include a data item and metadata of the data item, and the metadata of the data item includes an ID of the message and a cluster identifier of a cluster to which the producer of the data item belongs.
S108: based on the consumption cursor of the second replicator, messages are retrieved from a message queue corresponding to the target topic of the second cluster and saved to the second replicator.
S109: and calling a producer in the second duplicator to send the message stored in the second duplicator to the first cluster so that the target theme of the first cluster stores the message.
And under the condition that the message stored in the second duplicator does not contain the cluster identification of the first cluster, calling a producer in the second duplicator, and sending the message stored in the second duplicator to the first cluster so as to enable the target theme of the first cluster to store the message.
Optionally, when the message stored in the second replicator includes the cluster identifier of the first cluster, the message stored in the second replicator is prohibited from being sent to the first cluster, so that the message owned by the first cluster is prevented from being sent to the first cluster repeatedly.
S110: and after determining that the target theme of the first cluster successfully saves the message, sending ACK to the second duplicator so as to enable the second duplicator to update the consumption vernier of the second duplicator.
After the message is successfully sent to the first cluster, the producer of the first cluster shown in the second replicator needs to inform the second replicator of the successful message sending by the first cluster, so that the second replicator is prevented from wasting time and sending the message again, and the consumption vernier of the second replicator is updated in time so as to send other messages to the first cluster as soon as possible.
It should be noted that based on the flow shown in S101 to S110, message replication between multiple clusters can be implemented, for example, messages of the beijing cluster are replicated to the shanghai cluster and the shenzhen cluster, messages of the shanghai cluster are replicated to the beijing cluster and the shenzhen cluster, and messages of the shenzhen cluster are replicated to the beijing cluster and the shanghai cluster. Correspondingly, messages of a plurality of clusters can also be copied to the same cluster by unidirectional establishment, for example, messages of the Shenzhen cluster, messages of the Beijing cluster and messages of the Shanghai cluster are copied to the xi' an cluster.
In summary, compared with the prior art, in the embodiment, based on the first replicator preset in the first cluster and including the producer of the second cluster, the message included in the first cluster can be replicated to the second cluster without consuming machine resources of the first cluster, so that the message included in the cluster is replicated to other clusters without affecting the performance of the cluster, and the disaster tolerance capability of the cluster is effectively improved. In addition, because the first duplicator can be arranged in the browser node process of the first cluster, the cross-process and cross-network duplication loss among data can be eliminated, and the speed of duplicating the information of the first cluster to the second cluster is improved.
It should be noted that, in the above embodiment, the reference S101 is an optional implementation manner of the data processing method based on the message middleware in the present application. In addition, S110 mentioned in the above embodiment is also an optional implementation manner of the data processing method based on the message middleware described in this application. For this reason, the flow shown in the above embodiment can be summarized as the method shown in fig. 2.
As shown in fig. 2, a schematic flow chart of another data processing method based on message middleware provided in the embodiment of the present application includes the following steps:
s201: and saving the message produced by the producer of the first cluster into the target topic of the first cluster.
The message comprises a data item, and the target theme is a theme corresponding to the service type to which the data item belongs.
S202: based on the consumption cursor of the first replicator, messages are retrieved from a message queue corresponding to the target topic of the first cluster and saved to the first replicator.
Wherein the first replicator is pre-arranged in a first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster.
S203: and calling a producer in the first duplicator to send the message stored in the first duplicator to the second cluster so that the target theme of the second cluster stores the message.
S204: after determining that the target topic of the second cluster successfully saves the message, sending a confirmation character to the first replicator to cause the first replicator to update its own consumption curser.
In summary, compared with the prior art, in the embodiment, based on the first replicator preset in the first cluster and including the producer of the second cluster, the message included in the first cluster can be replicated to the second cluster without consuming machine resources of the first cluster, so that the message included in the cluster is replicated to other clusters without affecting the performance of the cluster, and the disaster tolerance capability of the cluster is effectively improved. In addition, because the first duplicator can be arranged in the brooker node process of the first cluster, the cross-process and cross-network duplication loss among data can be eliminated, and the speed of duplicating the information of the first cluster to the second cluster is improved.
Corresponding to the data processing method based on the message middleware provided by the embodiment of the application, the embodiment of the application also provides a data processing device based on the message middleware.
As shown in fig. 3, an architecture diagram of a data processing apparatus based on message middleware provided in an embodiment of the present application includes:
a message saving unit 100, configured to save a message generated by a producer of a first cluster into a target topic of the first cluster; the message comprises a data item; the target topic is a topic corresponding to the service type to which the data item belongs.
The message holding unit 100 is further configured to: and in case of failure of the first cluster, saving the message produced by the producer of the first cluster into the target topic of the second cluster.
The message holding unit 100 is further configured to: and under the condition that the second cluster does not fail, storing the message produced by the producer of the second cluster into the target theme of the second cluster.
In an embodiment of the application, the message further comprises metadata of the data item, the metadata comprising an ID of the message, a cluster identification of a cluster to which a producer of the data item belongs.
The message storage unit 100 is specifically configured to: in the event that the value of the ID of the message produced by the producer of the second cluster is not greater than the value of the ID of the message shown by the consumption cursor of the second replicator, saving the message to the target topic of the second cluster.
The message storage unit 100 is specifically configured to: and if the value of the ID of the message produced by the producer of the second cluster is larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, forbidding the message to be stored in the second cluster and deleting the message.
The message consumption unit 200 is used for acquiring a message from a message queue corresponding to the target topic of the first cluster based on the consumption cursor of the first duplicator, and storing the message into the first duplicator; the first duplicator is preset in a first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption cursor of the first replicator is pre-established based on the first replicator as a consumer of the first cluster.
The message consuming unit 200 is further configured to: based on the consumption cursor of the second duplicator, obtaining a message from a message queue corresponding to the target subject of the second cluster, and storing the message into the second duplicator; the second duplicator is preset in a second cluster and comprises a producer of the first cluster; the consumption cursors for the second replicator are pre-established based on the second replicator as consumers of the second cluster.
And the message copying unit 300 is used for calling a producer in the first duplicator and sending the message stored in the first duplicator to the second cluster so that the target theme of the second cluster stores the message.
The message copying unit 300 is further configured to: and calling a producer in the second duplicator to send the message stored in the second duplicator to the first cluster so that the target theme of the first cluster stores the message.
The message copying unit 300 is specifically configured to: and under the condition that the message stored in the second duplicator does not contain the cluster identification of the first cluster, calling a producer in the second duplicator, and sending the message stored in the second duplicator to the first cluster so as to enable the target theme of the first cluster to store the message.
The message copying unit 300 is specifically configured to: and under the condition that the message stored in the second duplicator contains the cluster identification of the first cluster, the message stored in the second duplicator is prohibited from being sent to the first cluster.
And the cursor updating unit 400 is used for sending a confirmation character to the first duplicator after determining that the target subject of the second cluster successfully stores the message, so that the first duplicator updates the own consumption cursor.
The cursor updating unit 400 is further configured to: after determining that the target topic of the first cluster successfully saves the message, sending a confirmation character to the second replicator to cause the second replicator to update its own consumption cursors.
In summary, compared with the prior art, in the embodiment, based on the first replicator preset in the first cluster and including the producer of the second cluster, the message included in the first cluster can be replicated to the second cluster without consuming machine resources of the first cluster, so that the message included in the cluster is replicated to other clusters without affecting the performance of the cluster, and the disaster tolerance capability of the cluster is effectively improved. In addition, because the first duplicator can be arranged in the browser node process of the first cluster, the cross-process and cross-network duplication loss among data can be eliminated, and the speed of duplicating the information of the first cluster to the second cluster is improved.
The application also provides a computer readable storage medium, which includes a stored program, wherein the program executes the message middleware-based data processing method provided by the application.
The present application further provides a data processing device based on message middleware, including: a processor, a memory, and a bus. The processor is connected with the memory through a bus, the memory is used for storing programs, and the processor is used for running the programs, wherein when the programs are run, the message middleware-based data processing method provided by the application is executed, and the message middleware-based data processing method comprises the following steps:
storing a message produced by a producer of a first cluster into a target topic of the first cluster; the message comprises a data item; the target theme is a theme corresponding to the business type to which the data item belongs;
based on a consumption cursor of a first replicator, obtaining the message from a message queue corresponding to a target topic of the first cluster, and saving the message to the first replicator; the first replicator is provisioned in the first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption vernier of the first duplicator is pre-established based on the first duplicator as a consumer of the first cluster;
calling a producer in the first duplicator to send the message stored in the first duplicator to the second cluster so that a target theme of the second cluster stores the message;
after determining that the target subject of the second cluster successfully saves the message, sending a confirmation character to the first replicator to enable the first replicator to update its own consumption cursors.
Specifically, on the basis of the above embodiment, the method further includes:
and under the condition that the first cluster fails, saving the message produced by the producer of the first cluster into the target theme of the second cluster.
Specifically, on the basis of the above embodiment, the method further includes:
under the condition that the second cluster does not fail, storing the message produced by the producer of the second cluster into a target theme of the second cluster;
based on a consumption cursor of a second replicator, obtaining the message from a message queue corresponding to a target topic of the second cluster, and saving the message to the second replicator; the second replicator is provisioned in the second cluster and comprises a producer of the first cluster; the consumption vernier of the second duplicator is pre-established based on the second duplicator as a consumer of the second cluster;
calling a producer in the second duplicator to send the message stored in the second duplicator to the first cluster so that a target subject of the first cluster stores the message;
after determining that the message is successfully saved by the target subject of the first cluster, sending a confirmation character to the second replicator to enable the second replicator to update its own consumption cursors.
Specifically, on the basis of the above embodiment, the message further includes metadata of a data item, the metadata including an ID of the message;
the saving the message produced by the producer of the second cluster to the target topic of the second cluster comprises:
and in the case that the value of the ID of the message produced by the producer of the second cluster is not larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, saving the message to the target subject of the second cluster.
Specifically, on the basis of the above embodiment, the method further includes:
and in the case that the value of the ID of the message produced by the producer of the second cluster is larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, forbidding the message to be stored in the second cluster and deleting the message.
Specifically, on the basis of the above embodiment, the message further includes metadata of the data item, where the metadata includes a cluster identifier of a cluster to which a producer of the data item belongs;
the calling a producer in the second duplicator to send the message saved in the second duplicator to the first cluster so that the target subject of the first cluster saves the message comprises:
and under the condition that the message stored in the second duplicator does not contain the cluster identification of the first cluster, calling a producer in the second duplicator, and sending the message stored in the second duplicator to the first cluster so as to enable the target subject of the first cluster to store the message.
Specifically, on the basis of the above embodiment, the method further includes:
and if the message stored in the second duplicator contains the cluster identifier of the first cluster, prohibiting the message stored in the second duplicator from being sent to the first cluster.
The functions described in the method of the embodiment of the present application, if implemented in the form of software functional units and sold or used as independent products, may be stored in a storage medium readable by a computing device. Based on such understanding, part of the contribution to the prior art of the embodiments of the present application or part of the technical solution may be embodied in the form of a software product stored in a storage medium and including several instructions for causing a computing device (which may be a personal computer, a server, a mobile computing device or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: u disk, removable hard disk, read only memory, random access memory, magnetic or optical disk, etc. for storing program codes.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A data processing method based on message middleware is characterized by comprising the following steps:
storing a message produced by a producer of a first cluster into a target topic of the first cluster; the message comprises a data item; the target theme is a theme corresponding to the service type to which the data item belongs;
based on a consumption cursor of a first replicator, obtaining the message from a message queue corresponding to a target topic of the first cluster, and saving the message to the first replicator; the first replicator is provisioned in the first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption vernier of the first duplicator is pre-established based on the first duplicator as a consumer of the first cluster;
calling a producer in the first duplicator to send the message stored in the first duplicator to the second cluster so that a target theme of the second cluster stores the message;
after determining that the target subject of the second cluster successfully saves the message, sending a confirmation character to the first replicator to enable the first replicator to update its own consumption cursors.
2. The method of claim 1, further comprising:
and under the condition that the first cluster fails, saving the message produced by the producer of the first cluster into the target theme of the second cluster.
3. The method of claim 1, further comprising:
under the condition that the second cluster does not fail, storing the message produced by the producer of the second cluster into a target theme of the second cluster;
based on a consumption cursor of a second replicator, obtaining the message from a message queue corresponding to a target topic of the second cluster, and saving the message to the second replicator; the second replicator is provisioned in the second cluster and comprises a producer of the first cluster; the consumption vernier of the second duplicator is pre-established based on the second duplicator as a consumer of the second cluster;
calling a producer in the second duplicator to send the message stored in the second duplicator to the first cluster so that a target subject of the first cluster stores the message;
after determining that the message is successfully saved by the target subject of the first cluster, sending a confirmation character to the second replicator to enable the second replicator to update its own consumption cursors.
4. The method of claim 3, wherein the message further comprises metadata for a data item, the metadata comprising an ID of the message;
the saving the message produced by the producer of the second cluster to the target topic of the second cluster comprises:
and in the case that the value of the ID of the message produced by the producer of the second cluster is not larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, saving the message to the target subject of the second cluster.
5. The method of claim 4, further comprising:
and in the case that the value of the ID of the message produced by the producer of the second cluster is larger than the value of the ID of the message shown by the consumption cursor of the second duplicator, forbidding the message to be stored in the second cluster and deleting the message.
6. The method of claim 3, wherein the message further comprises metadata for a data item, the metadata comprising a cluster identification of a cluster to which a producer of the data item belongs;
the calling a producer in the second duplicator to send the message saved in the second duplicator to the first cluster so that the target subject of the first cluster saves the message comprises:
and under the condition that the message stored in the second duplicator does not contain the cluster identification of the first cluster, calling a producer in the second duplicator, and sending the message stored in the second duplicator to the first cluster so as to enable the target subject of the first cluster to store the message.
7. The method of claim 6, further comprising:
and if the message stored in the second duplicator contains the cluster identifier of the first cluster, prohibiting the message stored in the second duplicator from being sent to the first cluster.
8. A message-middleware-based data processing apparatus, comprising:
the message storage unit is used for storing the message produced by the producer of the first cluster into the target theme of the first cluster; the message comprises a data item; the target theme is a theme corresponding to the service type to which the data item belongs;
the message consumption unit is used for acquiring the message from a message queue corresponding to the target subject of the first cluster based on the consumption cursor of the first duplicator and storing the message into the first duplicator; the first replicator is provisioned in the first cluster and comprises a producer of a second cluster; the first cluster and the second cluster belong to different areas respectively; the consumption vernier of the first duplicator is pre-established based on the first duplicator as a consumer of the first cluster;
a message replication unit, configured to invoke a producer in the first replicator, and send the message stored in the first replicator to the second cluster, so that a target topic of the second cluster stores the message;
and the cursor updating unit is used for sending a confirmation character to the first duplicator after the target theme of the second cluster is determined to successfully save the message, so that the first duplicator updates the self consumption cursor.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes the message middleware-based data processing method of any one of claims 1 to 7.
10. A message-middleware-based data processing apparatus, comprising: a processor, a memory, and a bus; the processor and the memory are connected through the bus;
the memory is used for storing a program, and the processor is used for executing the program, wherein the program executes the message middleware-based data processing method of any one of claims 1-7 in a running mode.
CN202210138474.7A 2022-02-15 2022-02-15 Data processing method and device based on message middleware Pending CN114513513A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210138474.7A CN114513513A (en) 2022-02-15 2022-02-15 Data processing method and device based on message middleware

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210138474.7A CN114513513A (en) 2022-02-15 2022-02-15 Data processing method and device based on message middleware

Publications (1)

Publication Number Publication Date
CN114513513A true CN114513513A (en) 2022-05-17

Family

ID=81551112

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210138474.7A Pending CN114513513A (en) 2022-02-15 2022-02-15 Data processing method and device based on message middleware

Country Status (1)

Country Link
CN (1) CN114513513A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106953901A (en) * 2017-03-10 2017-07-14 重庆邮电大学 A kind of trunked communication system and its method for improving message transmission performance
CN109391646A (en) * 2017-08-04 2019-02-26 中国电信股份有限公司 Message-oriented middleware message acquisition method, device and system
WO2020211344A1 (en) * 2019-04-17 2020-10-22 平安科技（深圳）有限公司 Mqtt-based message distribution method, server, apparatus, and storage medium
CN112527520A (en) * 2020-12-01 2021-03-19 中国建设银行股份有限公司 Method and device for deploying message middleware
CN112597249A (en) * 2020-12-26 2021-04-02 湖南快乐阳光互动娱乐传媒有限公司 Synchronous distribution and storage method and system for service data

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106953901A (en) * 2017-03-10 2017-07-14 重庆邮电大学 A kind of trunked communication system and its method for improving message transmission performance
CN109391646A (en) * 2017-08-04 2019-02-26 中国电信股份有限公司 Message-oriented middleware message acquisition method, device and system
WO2020211344A1 (en) * 2019-04-17 2020-10-22 平安科技（深圳）有限公司 Mqtt-based message distribution method, server, apparatus, and storage medium
CN112527520A (en) * 2020-12-01 2021-03-19 中国建设银行股份有限公司 Method and device for deploying message middleware
CN112597249A (en) * 2020-12-26 2021-04-02 湖南快乐阳光互动娱乐传媒有限公司 Synchronous distribution and storage method and system for service data

Similar Documents

Publication Publication Date Title
CN111368002A (en) Data processing method, system, computer equipment and storage medium
CN110535680B (en) Byzantine fault-tolerant method
CN101136728A (en) Cluster system and method for backing up a replica in a cluster system
JP2002522845A (en) Fault tolerant computer system
WO2013078689A1 (en) Method and device for realizing message transfer in cloud message service
CN110392120B (en) Method and device for recovering fault in message pushing process
CN105069152B (en) data processing method and device
WO2018010501A1 (en) Global transaction identifier (gtid) synchronization method, apparatus and system, and storage medium
CN111049928B (en) Data synchronization method, system, electronic device and computer readable storage medium
CN113641511A (en) Message communication method and device
CN105493474A (en) System and method for supporting partition level journaling for synchronizing data in a distributed data grid
CN113010496B (en) Data migration method, device, equipment and storage medium
CN113553179A (en) Distributed key value storage load balancing method and system
CN109582459A (en) The method and device that the trustship process of application is migrated
CN110635941A (en) Database node cluster fault migration method and device
CN112230853A (en) Storage capacity adjusting method, device, equipment and storage medium
CN110377664B (en) Data synchronization method, device, server and storage medium
CN112235405A (en) Distributed storage system and data delivery method
CN114124650A (en) Master-slave deployment method of SPTN (shortest Path bridging) network controller
CN111541762A (en) Data processing method, management server, device and storage medium
CN113867915A (en) Task scheduling method, electronic device and storage medium
CN112243030A (en) Data synchronization method, device, equipment and medium of distributed storage system
WO2024051454A1 (en) Method and apparatus for processing transaction log
CN113542314B (en) Method and system for realizing subscription mode based on keyword filtering
CN111752892B (en) Distributed file system and implementation method, management system, equipment and medium thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination