CN114064328A - Message queue cluster migration method and device - Google Patents
Message queue cluster migration method and device Download PDFInfo
- Publication number
- CN114064328A CN114064328A CN202111479614.9A CN202111479614A CN114064328A CN 114064328 A CN114064328 A CN 114064328A CN 202111479614 A CN202111479614 A CN 202111479614A CN 114064328 A CN114064328 A CN 114064328A
- Authority
- CN
- China
- Prior art keywords
- message
- cluster
- message queue
- queue cluster
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The scheme is used for establishing a second message queue cluster for all production side services in the system and only configuring one message theme. Each production side service generates a message carrying a message tag. And a mapping relation between the message labels and the message queues in the first message queue cluster is preset, and the messages in the second message queue cluster are forwarded to the corresponding message queues in the first message queue cluster based on the mapping relation. According to the scheme, different second message queue clusters and topics do not need to be created for different production side services, configuration workload is reduced, and migration efficiency is improved. Moreover, the mapping relations between different topics and the message queues in the first message queue cluster do not need to be configured respectively, so that the configuration workload is further reduced, and the migration efficiency is improved.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to a message queue cluster migration method and device.
Background
Message Queue (MQ) middleware refers to a container or service for storing messages in Message transmission, is an important component for realizing high-level special effects such as high performance, high availability and high scalability in a distributed system, and is applicable to various scenes, such as Message communication, asynchronous processing, application decoupling, traffic peak clipping and the like.
For example, two types of message queue middleware currently in the mainstream include ActiveMQ and rockmq. The ActiveMQ is an open-source Message queue middleware based on JMS (Java Message service) specification, the throughput of the ActiveMQ is ten thousand, and the availability is high. The RocktMQ is an open source message queue of Alibama, is developed by adopting Java language, has the characteristics of high throughput (10 ten thousand levels), very high availability and suitability for large-scale distributed system application.
However, considering the different performances of different message queue clusters, sometimes it is necessary to migrate a message queue cluster based on one message queue middleware to another message queue middleware, and a lot of configuration work is usually required, so that the migration efficiency is low and the error rate is high. For example, considering that rockmq has higher performance than ActiveMQ, it is necessary to migrate the service system originally based on the ActiveMQ cluster to the rockmq cluster to improve the performance of the entire service system.
Disclosure of Invention
In view of this, an object of the present invention is to provide a message queue cluster migration method and apparatus, so as to solve the problems of low efficiency and easy error in the message queue cluster migration process, and a specific technical solution thereof is as follows:
in a first aspect, the present application provides a message queue cluster migration method, which is applied in a message system, where the message system includes multiple production side services and multiple consumption side services, where a message queue cluster of the consumption side service is a first message queue cluster, and the method includes:
creating a second message queue cluster for the plurality of production side services, wherein the second message queue cluster is configured with a message subject;
generating a message carrying a message label by each production side service, and publishing the message to the message theme, wherein the message labels carried by the messages of different message types are different;
and sending each message in the second message queue cluster to a message queue matched with the message label of the message in the first message queue cluster according to a preset mapping relation between the message label and the message queue in the first message queue cluster.
In a possible implementation manner of the first aspect, the sending each message in the second message queue cluster to a message queue in the first message queue cluster, where the message queue is matched with a message tag of the message, according to a preset mapping relationship between the message tag and a message queue in the first message queue cluster includes:
analyzing the message in the message theme to obtain a target message label carried by the message;
inquiring the mapping relation between the message label and the message queue in the first message queue cluster to obtain a target message queue of the first message queue cluster matched with the target message label;
and sending the message to the target message queue.
In another possible implementation form of the first aspect, the message system comprises a common producer module;
the process of the production side service publishing the message to the message subject comprises the following steps:
and after any one of the production side services generates the message, calling a common producer module to send the message to the message theme of the second message queue cluster.
In yet another possible implementation manner of the first aspect, the first message queue cluster is an ActiveMQ cluster, and the second message queue cluster is a rockmq cluster.
In a second aspect, the present application further provides a message system, including multiple production side services, multiple consumption side services, and a message synchronization module, where a first message queue cluster corresponding to the consumption side service creates a second message queue cluster for the multiple production side services, and the second message queue cluster configures a message topic;
generating a message carrying a message label by each production side service, and publishing the message to the message theme, wherein the message labels carried by the messages of different message types are different;
and the message synchronization module is used for sending each message in the second message queue cluster to the message queue matched with the message label of the message in the first message queue cluster according to the preset mapping relation between the message label and the message queue in the first message queue cluster.
In one possible implementation of the second aspect, the message synchronization module includes a virtual consumer and a virtual producer;
the virtual consumer reads the message in the second message queue cluster and sends the message to the virtual producer;
the virtual producer analyzes the message to obtain a target message label carried by the message, inquires a preset mapping relation between the message label and a message queue in the first message queue cluster, obtains a target message queue in the first message queue cluster matched with the target message label, and issues the message to the target message queue.
In another possible implementation manner of the second aspect, each of the production-side services corresponds to a common producer module;
and calling the common producer module to issue the message produced by each production side service to the message subject of the second message queue cluster.
In yet another possible implementation manner of the second aspect, the first message queue cluster is an ActiveMQ cluster, and the second message queue cluster is a rockmq cluster.
In a third aspect, the present application further provides an apparatus, comprising: a memory and a processor, wherein the memory has instructions stored therein, and the processor executes the instructions to cause the device to perform the message queue cluster migration method of any one of the first aspect.
In a fourth aspect, the present application further provides a computer-readable storage medium, having stored thereon computer-executable instructions, which when loaded and executed by a processor, implement the message queue cluster migration method according to any one of the above first aspects.
In a fifth aspect, the present application further provides a computer program product adapted to, when executed on an electronic device, execute a program for initializing a message queue cluster migration method as described above in any of the first aspects.
According to the message queue cluster migration method provided by the application, a first message queue cluster is used by a consumption side service, a message queue cluster corresponding to each production side service is migrated from the first message queue cluster to a second message queue cluster, and messages generated by each production side service are all published to the same message theme (namely, Topic) of the second message queue cluster. Moreover, the message generated by each production side service carries a message tag. And sending the message issued by the production side service to a message queue matched with the message label according to the mapping relation between the preset message label and the message queue in the first message queue cluster, so that the consumption side service consumes the message based on the first message queue cluster. As can be seen from the above, in the scheme, all production side services share one second message queue cluster, and one message Topic is configured, so that different second message queue clusters and topics do not need to be created for different production side services, configuration workload is greatly reduced, and migration efficiency is improved. And the message generated by the production side service carries the message label, and each message is forwarded to the corresponding message queue in the first message queue cluster according to the mapping relation between the message label and the different message queues in the first message queue cluster, without respectively configuring the mapping relation between different Topic and the message queues in the first message queue cluster, thereby further reducing the configuration workload and improving the migration efficiency.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic diagram of a system structure based on ActiveMQ cluster;
fig. 2 is a schematic structural diagram of a message system according to an embodiment of the present application;
fig. 3 is a flowchart of a message queue cluster migration method according to an embodiment of the present application;
fig. 4 is a flowchart of a message forwarding process provided in an embodiment of the present application;
fig. 5 is a signaling diagram of another message queue cluster migration method according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Before describing the method provided by the present application in detail, the technical terms related to the present application or the related art will be described:
producer, responsible for generating messages, and general business systems, responsible for generating messages.
Consumer (message Consumer) is responsible for consuming messages, and general services are responsible for consuming messages.
Message is information to be transmitted, such as a Message generated by a production side service in the present application.
The Topic messaging model, under which a message publisher publishes a message to a particular Topic, 0 or more subscribers interested in the message Topic can receive the published message.
A Queue messaging model, by which a message producer can send a message to a message consumer, in which the message destination type is a Queue (Queue), the message is first transmitted to a message server-side specific Queue, from which it is then transmitted to the consumer listening to that Queue.
In one application scenario, the same system platform includes multiple different service subsystems, each of which can generate different types of messages, and messages generated by different producers are published to different message queue clusters. If the first message queue cluster used by each producer is migrated to the second message queue cluster. The consumer side still uses the first message queue cluster due to the large amount of traffic on the consumer side. The message transmission model of the first message queue cluster is different from that of the second message queue cluster, for example, the first message queue cluster adopts an ActiveMQ cluster based on a queue message transmission model, and the second message queue cluster adopts a rocktmq cluster based on a Topic message transmission model.
In this scenario, a second message queue cluster needs to be created for each producer, and meanwhile, in order to ensure that messages in the second message queue cluster can be consumed by consumers using the first message queue cluster, a mapping relationship between each type of message in each second message queue cluster and each message queue in the first message queue cluster needs to be configured, so that each type of message in the second message queue cluster can be forwarded to a corresponding message queue in the first message queue cluster on the consuming side for consumption by the consumers. However, when each first message queue cluster on the producer side is migrated to a second message queue cluster, a second message queue cluster needs to be created for each producer, and the workload of manual configuration is large, which results in low migration messages and is prone to errors. Meanwhile, the configuration workload of the mapping relationship between each type of message in the second message queue cluster and the message queue in the first message queue cluster is large, and further migration messages are low.
Taking the first message queue cluster as an ActiveMQ cluster as an example for explanation, please refer to fig. 1, which shows a schematic structural diagram of a messaging system based on the ActiveMQ cluster.
As shown in FIG. 1, the messaging system includes N producers, multiple ActiveMQ clusters, and multiple consumers. Wherein each ActiveMQ cluster comprises at least one VirtualTopic, each VirtualTopic comprising a plurality of message queues (queues).
The VirtualTopic is Topic for a producer and is Queue for a consumer, the internal processing mechanism is to distribute received messages to each Queue twice, and different consumers only need to care about and connect to own Queue to receive messages.
In view of the fact that the performance of an ActiveMQ cluster is lower than that of a rocktmq cluster, the ActiveMQ cluster of the service platform needs to be migrated to the rocktmq cluster. The service platform comprises a plurality of ActiveMQ clusters or a plurality of VirtualTopics, when the ActiveMQ clusters are migrated to the RockettMQ clusters, a plurality of RockettMQ clusters need to be created or a plurality of Topics need to be configured, the configuration workload is large, and problems are easy to occur. In addition, on the premise that the consuming end is not aware (that is, the consuming end still uses the ActiveMQ cluster to consume the messages), the ActiveMQ cluster of the producer is migrated to the rockmq cluster, and the corresponding relationship between each rockmq cluster or each Topic and the queue in the ActiveMQ cluster needs to be configured, so that the configuration workload is large and errors are prone to occurring.
In order to solve the problems in the cluster migration scenario, the application provides a message queue cluster migration method, which creates a second message queue cluster for all production side services in a system, and the second message queue cluster is configured with only one message topic. Each production side service generates a message carrying a message label, and the message labels corresponding to different types of messages are different. Moreover, a mapping relation between the message labels and the message queues in the first message queue cluster is preset, and the messages in the second message queue cluster can be forwarded to the corresponding message queues in the first message queue cluster based on the mapping relation. Therefore, different second message queue clusters and Topic do not need to be created for different production side services, configuration workload is greatly reduced, and migration efficiency is improved. And the message generated by the production side service carries the message label, and each message is forwarded to the corresponding message queue in the first message queue cluster according to the mapping relation between the message label and the different message queues in the first message queue cluster, without respectively configuring the mapping relation between different Topic and the message queues in the first message queue cluster, thereby further reducing the configuration workload and improving the migration efficiency.
Referring to fig. 2, a schematic structural diagram of a message system provided in an embodiment of the present application is shown, where the message system includes: n message producers, a second message queue cluster, a message synchronization module and a first message queue cluster.
For example, in a business system comprising a plurality of different businesses, a producer is the individual business that generates messages and a consumer is the service in the business system that receives the messages generated by the individual businesses.
In this embodiment, the first message queue cluster is an ActiveMQ cluster, and the second message queue cluster is a rockmq cluster, which are taken as examples for explanation.
And migrating the service at the producer side from the ActiveMQ cluster to a RockettMQ cluster, wherein a Topic is configured in the RockettMQ cluster, the message issued by each producer is issued to the Topic, and the message issued by the producer carries a message Tag.
Tag (message Tag) to provide additional flexibility to the user. Different types of messages generated by the same service module can be identified by different tags. The consumer can subscribe to and consume the message through Topic and Tag.
The message synchronization module can be deployed separately, and is used for forwarding the messages in the rocktmq cluster to a corresponding queue in the rocktmq cluster, so that the consumers consume the messages in the ActiveMQ cluster, and read the messages subscribed by themselves from the message queue in the ActiveMQ cluster for consumption, that is, the message cluster on the producer side migrates to the rocktmmq cluster and is unaware to the consumers.
In an exemplary embodiment, the message synchronization module includes a virtual consumer (i.e., a RockettMQ consumer) and a virtual producer (i.e., an ActiveMQ producer).
The virtual consumer, as a consumer of the second message queue cluster, reads the message in the Topic of the second message queue cluster and forwards the message to the virtual producer associated with the virtual consumer.
And the virtual producer is used as a producer of the first message queue cluster and used for issuing the message to the queue of the first message queue cluster, wherein the virtual producer issues the received message to the queue matched with the Tag carried by the message according to the mapping relation between the Tag and the queue.
In one possible implementation, to speed up message forwarding, multiple virtual consumers and multiple virtual producers are started simultaneously.
A plurality of virtual consumers can simultaneously consume the messages in the Topic of the RocktMQ cluster, and the message forwarding speed is accelerated.
In another possible implementation manner, each virtual consumer corresponds to multiple virtual producers, and the multiple virtual producers can process the received message simultaneously, that is, send the received message to the corresponding queue according to the mapping relationship between the Tag and the queue. The virtual producer with the highest publishing speed forwards the message to the queue matched with the message, and other virtual producers do not need to continuously process the message, so that the message forwarding rate can be increased by configuring a plurality of virtual producers.
In the message system provided in this embodiment, all production side services create a second message queue cluster, and the second message queue cluster is configured with only one message topic. Each production side service generates a message carrying a message label, and the message labels corresponding to different types of messages are different. Moreover, a mapping relation between the message labels and the message queues in the first message queue cluster is preset, and the messages in the second message queue cluster can be forwarded to the corresponding message queues in the first message queue cluster based on the mapping relation. Different second message queue clusters and Topic do not need to be created for different production side services, configuration workload is greatly reduced, and migration efficiency is improved. And forwarding each message to the corresponding message queue in the first message queue cluster according to the mapping relationship between the message label and the different message queues in the first message queue cluster, without configuring the mapping relationship between different Topic and the message queues in the first message queue cluster, thereby further reducing the configuration workload and improving the migration efficiency.
The message queue cluster migration process will be described in detail below with reference to a flow chart of a message synchronization method:
referring to fig. 3, a flowchart of a message Queue cluster migration method according to an embodiment of the present application is shown, and is applied to the message system shown in fig. 2, that is, a producer migrates from a first message Queue cluster to a second message Queue cluster, and a consumer still uses the first message Queue cluster, where the first message Queue cluster uses a Queue message transfer model and the second message Queue cluster uses a Topic message transfer model.
As shown in fig. 3, the method may include the steps of:
s110, a second message queue cluster is created, and the second message queue cluster is configured with a Topic.
In this embodiment, only one second message queue cluster needs to be created, and the second message queue cluster only configures one message Topic, i.e. Topic, to which all producers in the message system publish messages.
All producers issue messages to the same Topic, so all producers can share the same producer sending module, thereby avoiding repeatedly writing codes of different producer sending modules, and reducing the migration workload.
And S120, each producer generates a message carrying Tag and issues the message to the Topic.
Wherein, Tag represents the message type, i.e. tags in messages of different message types are different.
The tags in messages generated by different producers are different and the tags in messages of different message types generated by the same producer are different.
For example, the Tag corresponding to a suspense movie and television play generated by a video service in a video service platform is different from the Tag of an animation.
The Tag is a part of the message, and the producer can generate different tags according to the type of the message when producing the message, without additional manual creation or configuration.
S130, according to the mapping relation between the preset Tag and the message queue in the first message queue cluster, sending each message to the message queue matched with the Tag of the message in the first message queue cluster.
The mapping relationship between the Tag and the message queue in the first message queue cluster needs to be manually configured in advance. Before migration of the message clusters, the worker may configure a mapping relationship between the Tag and the Queue in the first message Queue cluster of the consumer according to the Queue in the first message Queue cluster of the producer side associated with each Queue in the first message Queue cluster of the consumer side, and a mapping relationship between the producer and the Queue in the first message Queue cluster of the producer side. The mapping relation is written into a configuration file, and the mapping relation can be obtained by reading the configuration file when the message system is initialized.
For example, a certain service system includes different types of service 1 and service 2, the two services generate different types of messages, the message generated by service 1 needs to be forwarded to queue1 of the ActiveMQ cluster, and the message generated by service 2 needs to be forwarded to queue2, in this case, the mapping relationship between Tag and queue is as follows: tag1 corresponds to queue1 and Tag2 corresponds to queue 2. When the message generated by the producer needs to be forwarded to the consumer side, the message containing the Tag1 is forwarded to the queue1 according to the mapping relation, and the message containing the Tag2 is forwarded to the queue 2.
In an exemplary embodiment, as shown in fig. 4, the process shown in S130 may include the steps of:
s131, analyzing the message in the message theme to obtain the target message label carried by the message.
And reading the message in the Topic of the second message queue cluster, and analyzing the Tag carried in the message, namely the target message Tag.
S132, inquiring the mapping relation between the message labels and the message queues in the first message queue cluster, and obtaining the target message queue of the first message queue cluster matched with the target message labels.
And inquiring the mapping relation between the Tag and the queue to obtain the queue matched with the target message label in the first message queue cluster, namely the target message queue.
S133, sending the message to the target message queue.
And finally, sending the message read from the Topic of the second message queue cluster to a target message queue in the first message queue cluster, and finally forwarding the message in the second message queue cluster to the first message queue cluster so that the consumer can consume the message according to the queue message consumption mode.
In one embodiment of the present application, S130 may be performed by a message synchronization module in the messaging system shown in fig. 2, the message synchronization module comprising virtual consumers and virtual producers, wherein each virtual consumer is associated with at least one virtual producer.
The virtual consumer, as a consumer of the second message queue cluster, reads the message in the Topic of the second message queue cluster and forwards the message to the virtual producer associated with the virtual consumer.
And the virtual producer is used as a producer of the first message queue cluster and used for issuing the message to the queue of the first message queue cluster, wherein the virtual producer issues the received message to the queue matched with the Tag carried by the message according to the mapping relation between the Tag and the queue.
In the embodiment, only the mapping relation between each Tag and each queue needs to be configured, and Topic corresponding to the queue on the consumer side one by one does not need to be configured on the producer side, so that the workload of manual configuration is further reduced.
In the message queue cluster migration method provided in this embodiment, a second message queue cluster is created for all production side services in the system, and only one message topic is configured for the second message queue cluster. Each production side service generates a message carrying a message label, and the message labels corresponding to different types of messages are different. Moreover, a mapping relation between the message labels and the message queues in the first message queue cluster is preset, and the messages in the second message queue cluster can be forwarded to the corresponding message queues in the first message queue cluster based on the mapping relation. Therefore, different second message queue clusters and Topic do not need to be created for different production side services, configuration workload is greatly reduced, and migration efficiency is improved. And the message generated by the production side service carries the message label, and each message is forwarded to the corresponding message queue in the first message queue cluster according to the mapping relation between the message label and the different message queues in the first message queue cluster, without configuring the mapping relation between different Topic and the message queues in the first message queue cluster, thereby further reducing the configuration workload and improving the migration efficiency.
Referring to fig. 5, a flowchart of another message queue cluster migration method provided in an embodiment of the present application is shown, where a first message queue cluster in the embodiment is an ActiveMQ cluster based on a topoic messaging model, and a second message queue cluster is a rockmq cluster based on a queue messaging model.
As shown in fig. 5, the method may include the steps of:
s210, a RocketMQ cluster is created, and the RocketMQ cluster is configured with a Topic.
The messaging system shown in fig. 2 includes N message producers (i.e., the business parties that generate the messages), which all publish to the Topic. The sending modules of the N producers execute the same logic for sending messages to the same Topic, so that the N producers share one common producer module, and after any producer generates a message, the common producer module is called to send the message to the same Topic. Therefore, the code of the producer module can be prevented from being repeatedly written, and the workload of developers is reduced.
In an exemplary embodiment, the producer modules of the rockmq cluster can be dynamically loaded in a modularized manner by each production-side service, that is, whether the configuration information of the instantiation producer is configured by the corresponding configuration information is judged, if the configuration information of the instantiation producer is configured, the instantiation producer is required, and each production-side service is not required to instantiate the producer, so that the system resource overhead is reduced.
In other words, the producer program does not need to be configured for each production side business, and only the producer program needs to be developed for any one production side business, so that the repeated development of the producer program is avoided, and the workload of developers is reduced.
S220, the production side service generates a message carrying the Tag, and the tags corresponding to the messages of different message types are different.
In an exemplary embodiment, corresponding tags may be configured for message types in the entire service system, where the tags corresponding to different message types are different, for example, a message Tag corresponding to a type a message is Tag1, and a message Tag corresponding to a type B message is Tag 2.
When the production side service generates a certain type of message, the Tag corresponding to the type is carried in the message.
Further, configuring the mapping relation between the Tag and the queue in the ActiveMQ cluster, and writing the mapping relation into the configuration file.
And S230, the production side service utilizes the common producer module to issue the message to the Topic of the RocktetMQ cluster.
After any production side service generates a message, a common producer module is started, and the message is issued to the Topic through the common producer module.
S240, the virtual consumer in the message synchronization module reads the message from the Topic and forwards the message to the virtual producer associated with the virtual consumer.
The message synchronization module includes a virtual consumer and a virtual producer, wherein the virtual consumer is a consumer of the rocktmq cluster and is configured to read messages in the rocktmq cluster. The virtual producer is the producer of the ActiveMQ cluster, and is used for issuing messages to the queue of the ActiveMQ cluster.
In one embodiment of the present application, to increase message forwarding speed, multiple virtual consumers and multiple virtual producers are started simultaneously.
A plurality of virtual consumers can simultaneously consume the messages in the Topic of the RocktMQ cluster, and the message forwarding speed is accelerated.
In an exemplary embodiment, one virtual consumer and n virtual producers are started for each set of mapping relationships according to the mapping relationships between Tag and queue in the configuration file, and n is greater than or equal to 1.
And each virtual consumer sends the read message to all the virtual producers related to the virtual consumer, so that a plurality of virtual producers can process the received message simultaneously, namely, the received message is sent to the corresponding queue according to the mapping relation between Tag and queue.
The virtual producer with the highest publishing speed forwards the message to the queue matched with the message, and other virtual producers do not need to continuously process the message, so that the message forwarding rate can be increased by configuring a plurality of virtual producers.
And S250, the virtual producer analyzes the target message label carried by the message, and determines a target message queue matched with the target message label according to the mapping relation between the Tag and the queue.
When the message synchronization module is initialized, the mapping relation between Tag and queue in the configuration file is read, after the virtual producer receives the message, the virtual producer analyzes the message to obtain a target message label in the message, and queries a target message queue matched with the target label based on the mapping relation.
And S260, the virtual producer publishes the received message to a target message queue.
And the virtual producer publishes the received message to a target message queue of the ActiveMQ cluster based on the mapping relation between the Tag and the queue.
For example, tag a corresponds to queue1, and the virtual producer receives the message containing tag a and sends the message to queue1 of the ActiveMQ cluster. For another example, tag b corresponds to queue2, and when the virtual producer receives a message containing tag b, the message is forwarded to queue 2.
S270, the service at the consumption side reads the subscribed information from the queue of the ActiveMQ cluster.
The consuming side service can consume the message directly according to the message consumption mode of the ActiveMQ cluster, for example, the consumer 1 subscribes to the message in the queue1, the consumer 1 listens to the queue1, and when the message is listened to in the queue1, the consumer 1 reads the message from the queue1 directly.
According to the message queue cluster migration method provided by the embodiment, different RocktMQ clusters and Topic do not need to be created for different production side services, so that the configuration workload is greatly reduced, and the migration efficiency is improved. And forwarding each message to a corresponding message queue in the first message queue cluster according to the mapping relationship between the message label and different message queues in the first message queue cluster, without configuring the mapping relationship between different Topic and the message queues in the first message queue cluster, thereby further reducing the configuration workload and improving the migration efficiency. Moreover, a plurality of rockemq consumers (i.e., virtual consumers) and a plurality of ActiveMQ producers (i.e., virtual producers) are configured in the message synchronization module in this embodiment, the plurality of virtual consumers forward the same message in the Topic to the corresponding virtual producer, each virtual producer issues the message to a queue in the ActiveMQ cluster based on the mapping relationship between Tag and the queue, and the plurality of virtual consumers and the virtual producers process one message at the same time, thereby accelerating the message forwarding rate and improving the message forwarding efficiency.
The application also provides a storage medium executable by a computing device, wherein the storage medium stores a program, and the program realizes the message queue cluster migration method when executed by the computing device.
The present application further provides a computer program product adapted to perform a program for initializing steps of a message queue cluster migration method of any of the above when executed on an electronic device.
While, for purposes of simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present invention is not limited by the illustrated ordering of acts, as some steps may occur in other orders or concurrently with other steps in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
It should be noted that technical features described in the embodiments in the present specification may be replaced or combined with each other, each embodiment is mainly described as a difference from the other embodiments, and the same and similar parts between the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The steps in the method of the embodiments of the present application may be sequentially adjusted, combined, and deleted according to actual needs.
The device and the modules and sub-modules in the terminal in the embodiments of the present application can be combined, divided and deleted according to actual needs.
In the several embodiments provided in the present application, it should be understood that the disclosed terminal, apparatus and method may be implemented in other manners. For example, the above-described terminal embodiments are merely illustrative, and for example, the division of a module or a sub-module is only one logical division, and there may be other divisions when the terminal is actually implemented, for example, a plurality of sub-modules or modules may be combined or integrated into another module, or some features may be omitted or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules or sub-modules described as separate parts may or may not be physically separate, and parts that are modules or sub-modules may or may not be physical modules or sub-modules, may be located in one place, or may be distributed over a plurality of network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, each functional module or sub-module in the embodiments of the present application may be integrated into one processing module, or each module or sub-module may exist alone physically, or two or more modules or sub-modules may be integrated into one module. The integrated modules or sub-modules may be implemented in the form of hardware, or may be implemented in the form of software functional modules or sub-modules.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A message queue cluster migration method is applied to a message system, wherein the message system comprises a plurality of production side services and a plurality of consumption side services, and a message queue cluster of the consumption side service is a first message queue cluster, and the method comprises:
creating a second message queue cluster for the plurality of production side services, wherein the second message queue cluster is configured with a message subject;
generating a message carrying a message label by each production side service, and publishing the message to the message theme, wherein the message labels carried by the messages of different message types are different;
and sending each message in the second message queue cluster to a message queue matched with the message label of the message in the first message queue cluster according to a preset mapping relation between the message label and the message queue in the first message queue cluster.
2. The method of claim 1, wherein the sending each message in the second message queue cluster to the message queue in the first message queue cluster matching the message tag of the message according to a preset mapping relationship between the message tag and the message queue in the first message queue cluster comprises:
analyzing the message in the message theme to obtain a target message label carried by the message;
inquiring the mapping relation between the message label and the message queue in the first message queue cluster to obtain a target message queue of the first message queue cluster matched with the target message label;
and sending the message to the target message queue.
3. The method of claim 1, wherein the messaging system includes a common producer module;
the process of the production side service publishing the message to the message subject comprises the following steps:
and after any one of the production side services generates the message, calling a common producer module to send the message to the message theme of the second message queue cluster.
4. The method of any of claims 1-3, wherein the first message queue cluster is an ActiveMQ cluster and the second message queue cluster is a RockcketMQ cluster.
5. A message system is characterized by comprising a plurality of production side services, a plurality of consumption side services and a message synchronization module, wherein a first message queue cluster corresponding to the consumption side services establishes a second message queue cluster for the plurality of production side services, and the second message queue cluster is configured with a message theme;
generating a message carrying a message label by each production side service, and publishing the message to the message theme, wherein the message labels carried by the messages of different message types are different;
and the message synchronization module is used for sending each message in the second message queue cluster to the message queue matched with the message label of the message in the first message queue cluster according to the preset mapping relation between the message label and the message queue in the first message queue cluster.
6. The messaging system of claim 5, wherein the message synchronization module comprises a virtual consumer and a virtual producer;
the virtual consumer reads the message in the second message queue cluster and sends the message to the virtual producer;
the virtual producer analyzes the message to obtain a target message label carried by the message, inquires a preset mapping relation between the message label and a message queue in the first message queue cluster, obtains a target message queue in the first message queue cluster matched with the target message label, and issues the message to the target message queue.
7. The messaging system of claim 5, wherein each of the production-side services corresponds to a common producer module;
and calling the common producer module to issue the message produced by each production side service to the message subject of the second message queue cluster.
8. The messaging system of claim 5, wherein the first message queue cluster is an ActiveMQ cluster and the second message queue cluster is a RockcketMQ cluster.
9. An apparatus, comprising: a memory having stored therein instructions, and a processor executing the instructions to cause the apparatus to perform the message queue cluster migration method of any of claims 1-4.
10. A computer-readable storage medium having computer-executable instructions stored thereon which, when loaded and executed by a processor, implement the message queue cluster migration method of any one of claims 1-4 above.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111479614.9A CN114064328A (en) | 2021-12-06 | 2021-12-06 | Message queue cluster migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111479614.9A CN114064328A (en) | 2021-12-06 | 2021-12-06 | Message queue cluster migration method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114064328A true CN114064328A (en) | 2022-02-18 |
Family
ID=80228673
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111479614.9A Pending CN114064328A (en) | 2021-12-06 | 2021-12-06 | Message queue cluster migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114064328A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114979257A (en) * | 2022-05-24 | 2022-08-30 | 树根互联股份有限公司 | Message pushing method and device, computer equipment and computer readable storage medium |
| CN115086400A (en) * | 2022-06-20 | 2022-09-20 | 平安普惠企业管理有限公司 | Message interaction method and device, electronic equipment and storage medium |
-
2021
- 2021-12-06 CN CN202111479614.9A patent/CN114064328A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114979257A (en) * | 2022-05-24 | 2022-08-30 | 树根互联股份有限公司 | Message pushing method and device, computer equipment and computer readable storage medium |
| CN115086400A (en) * | 2022-06-20 | 2022-09-20 | 平安普惠企业管理有限公司 | Message interaction method and device, electronic equipment and storage medium |
| CN115086400B (en) * | 2022-06-20 | 2023-11-10 | 光禹莱特数字科技(上海)有限公司 | Message interaction method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111290854B (en) | Task management method, device, system, computer storage medium and electronic equipment | |
| US20200379812A1 (en) | Unified container orchestration controller | |
| US9882844B2 (en) | Opaque message parsing | |
| US10700948B2 (en) | Service-oriented modular system architecture | |
| CN114064328A (en) | Message queue cluster migration method and device | |
| CN109995801B (en) | Message transmission method and device | |
| CN112069265A (en) | Configuration data synchronization method, service data system, computer system and medium | |
| CN111966943A (en) | Streaming data distribution method and system | |
| CN111131034B (en) | Label-based environment isolation method for distributed system | |
| Sabbioni et al. | DIFFUSE: A DIstributed and decentralized platForm enabling Function composition in Serverless Environments | |
| CN115268949A (en) | Mirror preheating method, device, equipment and storage medium | |
| US8812578B2 (en) | Establishing future start times for jobs to be executed in a multi-cluster environment | |
| CN113220432A (en) | Multi-cloud interconnection method, device, equipment, storage medium and product | |
| CN113743879A (en) | Automatic rule processing method, system and related equipment | |
| CN111951040A (en) | Information sending method and device, electronic equipment and storage medium | |
| CN110673827A (en) | Resource calling method and device based on android system and electronic equipment | |
| US20100332604A1 (en) | Message selector-chaining | |
| CN116107710A (en) | Method, apparatus, device and medium for processing offline rendering tasks | |
| CN115378937A (en) | Distributed concurrency method, device and equipment for tasks and readable storage medium | |
| CN113254143B (en) | Virtualized network function network element arrangement scheduling method, device and system | |
| CN110995725B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN113098960A (en) | Service operation method, device, server and storage medium | |
| CN115442420A (en) | Block chain cross-chain service management method and device | |
| CN113296968A (en) | Address list updating method, device, medium and electronic equipment | |
| CN114936098B (en) | Data transfer method, device, back-end equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |