CN116319591A - Data processing method of message queue cluster, electronic equipment and storage medium - Google Patents

Data processing method of message queue cluster, electronic equipment and storage medium Download PDF

Info

Publication number
CN116319591A
CN116319591A CN202211553290.3A CN202211553290A CN116319591A CN 116319591 A CN116319591 A CN 116319591A CN 202211553290 A CN202211553290 A CN 202211553290A CN 116319591 A CN116319591 A CN 116319591A
Authority
CN
China
Prior art keywords
message queue
configuration
service
configuration information
data source
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211553290.3A
Other languages
Chinese (zh)
Inventor
钱玉森
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Bank Co Ltd
Original Assignee
Ping An Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Bank Co Ltd filed Critical Ping An Bank Co Ltd
Priority to CN202211553290.3A priority Critical patent/CN116319591A/en
Publication of CN116319591A publication Critical patent/CN116319591A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/12—Avoiding congestion; Recovering from congestion
    • H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/70—Admission control; Resource allocation

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application discloses a data processing method, electronic equipment and storage medium of a message queue cluster, wherein the data processing method of the message queue cluster comprises the following steps: responding to the received configuration change notification sent by the message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service; the control message queue data source service performs resource configuration on the message queues corresponding to the configuration information so as to balance the loads of a plurality of message queues in the message queue cluster. By means of the method, the data processing method for the message queue cluster can achieve automatic resource allocation of the message queues, and therefore the problems that the message queues are unbalanced in load and cannot be started are avoided.

Description

Data processing method of message queue cluster, electronic equipment and storage medium
Technical Field
The application relates to the technical field of computers, in particular to a data processing method of a message queue cluster, electronic equipment and a storage medium.
Background
Some distributed systems often need to access a plurality of message queues, but the information amounts stored in different message queues are different, and in the actual application process, the message queues with large information amounts are found to be easy to start, and the load of the message queues contained in the whole message queue cluster is unbalanced.
Disclosure of Invention
The self-request provides a data processing method, electronic equipment and storage medium of a message queue cluster, which can realize resource allocation of message queues in the message queue cluster so as to realize message queue load balancing.
In order to solve the technical problems, one technical scheme adopted by the application is as follows: there is provided a data processing method of a message queue cluster, wherein the message queue cluster comprises a plurality of message queues of different types, the method comprising: responding to the received configuration change notification sent by the message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service; the control message queue data source service performs resource configuration on the message queues corresponding to the configuration information so as to balance the loads of a plurality of message queues in the message queue cluster.
The control message queue data source service performs resource configuration on the message queue corresponding to the configuration information, and the control message queue data source service comprises: acquiring a corresponding server identifier in the configuration information; determining a target message queue from a plurality of message queues according to the server identification; the control message queue data source service performs resource configuration on the target message queue according to the configuration information.
Wherein the resource allocation includes one of deleting a resource, adding a resource, and modifying a resource.
Wherein the configuration information includes a theme and a subtopic.
The control message queue data source service performs resource configuration on the target message queue according to the configuration information, and comprises the following steps: the control message queue data source service determines a target data source corresponding to the target message queue; and establishing the corresponding relation between the target data source and the topics and the subtopics.
Wherein the configuration change notification is a newly added message queue notification; in response to receiving a configuration change notification sent by the message queue configuration service, obtaining corresponding configuration information from the message queue configuration service, including: and responding to the received notification of the newly added message queue sent by the message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service.
In addition, the control message queue data source service performs resource configuration on the message queue corresponding to the configuration information, and the control message queue data source service comprises: and the control message queue data source service adds a message queue according to the configuration information and performs resource configuration on the newly added message queue.
Wherein the configuration information includes message queue type, topic, sub-topic and server identification.
Wherein before receiving the configuration change notification sent by the message queue configuration service, the method comprises the following steps: and detecting that the configuration information in the message queue configuration service is changed, and generating a configuration change notification.
Wherein, the method for generating the configuration change notification comprises the following steps of: the method comprises the steps of regularly monitoring that configuration information in a message queue configuration service is changed, and generating a configuration change notification; or, the configuration information in the message queue configuration service is monitored to be changed in real time, and a configuration change notification is generated.
In order to solve the technical problem, another technical scheme adopted by the application is as follows: there is provided an electronic device comprising a memory for storing a computer program and a processor for executing the computer program to implement the data processing method of a message queue cluster as described above.
In order to solve the technical problem, another technical scheme adopted by the application is as follows: there is provided a computer readable storage medium for storing a computer program which, when executed by a processor, is adapted to carry out the data processing method of a message queue cluster as described above.
The beneficial effects of this application are: compared with the prior art, the data processing method of the message queue cluster provided by the application is characterized in that corresponding configuration information is obtained from the message queue configuration service by responding to the configuration change notification sent by the message queue configuration service, so that the message queue data source service is controlled to carry out resource configuration on the message queues corresponding to the configuration information, so that the load balance of a plurality of message queues in the message queue cluster can be realized, the resource of the message queues can be dynamically adjusted, the condition of unbalanced load of the message queues is avoided, and the resource in the message queues can be redistributed when unbalanced load occurs.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a flowchart of a first embodiment of a message queue cluster data processing method provided in the present application;
FIG. 2 is a flowchart of a first embodiment of a resource allocation method provided in the present application;
FIG. 3 is a flowchart of a second embodiment of a resource allocation method provided in the present application;
FIG. 4 is a flowchart of a second embodiment of a message queue cluster data processing method provided in the present application;
FIG. 5 is a flowchart of a third embodiment of a message queue cluster data processing method provided in the present application;
FIG. 6 is a schematic diagram illustrating one embodiment of a message queue cluster data processing method provided herein;
FIG. 7 is a schematic structural diagram of an embodiment of an electronic device provided herein;
fig. 8 is a schematic structural diagram of an embodiment of a computer readable storage medium provided in the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Referring to fig. 1, fig. 1 is a flowchart of a first embodiment of a message queue cluster data processing method provided in the present application, where a message queue cluster includes a plurality of message queues of different types, and the method includes:
step 11: and responding to the received configuration change notification sent by the message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service.
The message queue is MQ (Message Queue), which is a container for storing messages in the process of message transmission, can utilize an efficient and reliable message transmission mechanism to perform platform-independent data communication, and is based on data communication for integrating a distributed system.
In some embodiments, the message queue configuration service is an MQ configuration service that supports parameter, resource configuration for any of the message queues in the message queue cluster.
The change notification refers to a notification triggered by a change of configuration information of a corresponding message queue in the message queue configuration service. The reason for changing the configuration information may be that the number of resources of the message queue exceeds a threshold value, so as to better manage the resources of the message queue and correspondingly change the configuration information; or the example of the message queue can not be started, and the configuration information is changed correspondingly for starting the example; the configuration information may also be changed accordingly in response to a user's manipulation of the message queue.
It will be appreciated that the message queues in which the configuration change occurs include at least one, and that when the configuration change occurs simultaneously in a plurality of message queues, corresponding change notifications may be generated and sent synchronously.
In some embodiments, when the message queue configuration service monitors that the configuration information of the message queue is changed, a change notification is triggered, and the configuration change notification is sent to the message queue data source service, so as to control the message queue data source service to perform corresponding operations, for example, control the message queue data source service to perform information query on the message queue configuration service, and acquire the configuration information corresponding to the configuration change notification.
Step 12: the control message queue data source service performs resource configuration on the message queues corresponding to the configuration information so as to balance the loads of a plurality of message queues in the message queue cluster.
In some embodiments, the message queue data source service is an MQ data source service, and when receiving the configuration change notification sent by the message queue configuration service, the MQ data source service may obtain configuration information corresponding to the configuration change notification from the message queue configuration service, and then configure resources of the corresponding message queue according to the configuration information.
Wherein the resource allocation includes one of deleting a resource, adding a resource, and modifying a resource. The resources of the message queue include at least topics and consumed messages.
In some embodiments, the configuration information includes a server identifier, and referring to fig. 2, the resource configuration of the message queue based on the configuration information may include the following procedures:
step 21: and obtaining a corresponding server identifier in the configuration information.
The server identification may be an IP address of the server, or other information that may distinguish between different servers, such as a MAC address of the server, without limitation.
Step 22: a target message queue is determined from the number of message queues based on the server identification.
In some embodiments, the message queue data source service may be controlled to invoke a number of message queues according to the IP address of the server to determine a target message queue from the number of message queues.
In some embodiments, the target message server may be located via an external interface, such as an API (Application Programming Interface ), based on the server's identification information.
Step 23: the control message queue data source service performs resource configuration on the target message queue according to the configuration information.
In particular, the message queue data source service may be controlled to delete, add, and/or modify resources of the target message queue. For example, a portion of the resources in the target message queue may be deleted, a portion of the resources in the target message queue may be modified, such as modifying resource a to resource B, or a new portion of the resources C in the target message queue.
In some embodiments, the configuration information includes a Topic (Topic) and a subtopic (Tag), and referring to fig. 3, the resource configuration of the message queue based on the configuration information may include the following procedures:
step 31: the control message queue data source service determines a target data source corresponding to the target message queue.
In particular, the control message queue data source service may determine a target data source based on the topic and the sub-topic. The target data source may be one of an ISC (Integrated Supply Chain ) data source, a dockmq data source, and a Kafka data source.
Step 32: and establishing the corresponding relation between the target data source and the topics and the subtopics.
Specifically, the target data source contains several messages (messages), each of which must belong to a topic. In addition, the messages in the target data source all have a unique identifier, namely a Message ID (Message name), and the Message ID can carry a Key (Key) with a service identifier.
The same topic contains different types of messages, and different sub-topics may be provided for the same topic in order to distinguish between different messages from the same topic, e.g. topic a comprises sub-topics B, C.
In summary, the configuration information of the present application includes at least a message queue type, a topic, a sub-topic, and a server identification.
Compared with the prior art, the message queue cluster data processing method provided by the application can acquire corresponding configuration information from the message queue configuration service by responding to the configuration change notification sent by the message queue configuration service, so as to control the message queue data source service to carry out resource configuration on the message queues corresponding to the configuration information, thereby realizing dynamic adjustment of the resources of the message queues, further balancing the loads of a plurality of message queues in the message queue cluster, effectively avoiding the condition of unbalanced loads of the message queues, and reallocating the resources in the message queues when unbalanced loads occur.
Referring to fig. 4, fig. 4 is a flowchart of a second embodiment of a message queue cluster data processing method provided in the present application, where a message queue cluster includes a plurality of message queues of different types, and the method includes:
step 41: and detecting that the configuration information in the message queue configuration service is changed, and generating a configuration change notification.
In some embodiments, configuration information in the message queue configuration service may be monitored periodically to monitor whether a change has occurred to the configuration information in the message queue configuration service and to generate a configuration change notification when a change has occurred.
In other embodiments, configuration information in the message queue configuration service may be monitored in real-time to monitor whether the configuration information in the message queue configuration service changes, and generate a configuration change notification when a change occurs.
Step 42: and responding to the received configuration change notification sent by the message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service.
Step 43: the control message queue data source service performs resource configuration on the message queues corresponding to the configuration information so as to balance the loads of a plurality of message queues in the message queue cluster.
Steps 42 to 43 may have the same or similar technical features as any of the above embodiments, and will not be described herein.
Compared with the prior art, the message queue cluster data processing method provided by the application can control the message queue data source service to carry out resource configuration on the message queue with changed configuration information by monitoring the configuration information in the message queue configuration service, so that the problem of unbalanced load of the message queue is avoided.
In other embodiments, the configuration change notification is a newly added message queue notification. Referring now to fig. 5, fig. 5 is a flow chart of a third embodiment of a message queue cluster data processing method provided in the present application, where a message queue cluster includes a plurality of message queues of different types, and the method includes:
step 51: and responding to the received notification of the newly added message queue sent by the message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service.
It will be appreciated that the configuration change notification includes at least one of a new message queue notification, a modified message queue notification, and a deleted message queue notification. Different notifications may be sent simultaneously based on traffic demand.
When receiving the notification of the newly added message queue sent by the message queue configuration service, the corresponding configuration information can be queried through the message queue data source control service. The message queue data source control service is used for controlling the message queue data source service to perform corresponding operations, such as adding and deleting.
Step 52: and controlling the message queue data source service to newly add a message queue according to the configuration information, and carrying out resource configuration on the newly added message queue so as to load balance a plurality of message queues in the message queue cluster.
In some embodiments, when the resources in the message queue exceed the threshold, the problems of unbalanced load of the message queue, incapability of starting an instance in the message queue, and the like may be caused, and at this time, some message queues of the same type may be newly added in the message queue cluster, so that the newly added message queue resources are used to store the resources that cannot be stored in other message queues, so that the loads of a plurality of message queues in the message queue cluster are balanced.
Compared with the prior art, the message queue cluster data processing method provided by the application can realize newly added message queues so as to utilize the newly added message queues to receive partial resources of other message queues with unbalanced loads and realize load balancing of all message queues in the message queue cluster.
Referring to fig. 6, fig. 6 is a schematic diagram of an embodiment of a message queue cluster data processing method provided in the present application, where an MQ configuration service corresponds to a message queue configuration service, an MQ data source control service corresponds to a message queue data source control service, and an MQ data source service corresponds to a message queue data source service.
The MQ configuration service may send a configuration update notification to the MQ data source control service, so that the MQ data source control service queries and obtains corresponding configuration information, and further, drives/controls the MQ data source service to perform resource configuration on the message queue by using the MQ data source control service, such as deleting resources and adding resources.
The MQ data source service may update the ISC data source, the dockmq data source, and/or the Kafka data source and store the resources of the altered message queue to the corresponding data source. The message queue in the ISC data source may include a topic I1 and a topic I3, where the sub-topic of I1 is IT1 and the sub-topic of I3 is IT2; the message queue in the RocketMQ data source may include a topic R1 and a topic R2, where the sub-topic of R1 is RT1 and the sub-topic of R2 is RT2; the message queue in the Kafka data source may include a topic K1 and a topic K2, where K1 is a sub-topic of KT1 and K2 is a sub-topic of KT2.
In other embodiments, the MQ configuration services include a user configuration service, a configuration query service, and a configuration change detection service. The user configuration service means that the user can configure the resources of the message queue, such as adding, deleting, modifying, inquiring and the like; the configuration query service refers to acquiring configuration information through an external interface based on IP address information, such as acquiring configuration information of a target message queue through an API; the configuration change detection service can detect configuration information of message queues in the message queue cluster at regular time or in real time, so as to trigger corresponding change notification when the configuration information is detected to be changed.
Referring to fig. 7, fig. 7 is a schematic structural diagram of an embodiment of an electronic device provided in the present application, where the electronic device 70 includes a memory 701 and a processor 702, the memory 701 is configured to store a computer program, and the processor 702 is configured to execute the computer program to implement a message queue cluster data processing method according to any one of the embodiments described above, which is not described herein.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an embodiment of a computer readable storage medium provided in the present application, where a computer program 801 is stored in the computer readable storage medium 80, and the computer program 801, when executed by a processor, is used to implement a message queue cluster data processing method of any one of the foregoing embodiments, which is not described herein again.
In summary, the message queue cluster data processing method provided by the application can realize dynamic management of the resources of the message queue, avoid the condition of unbalanced load of the message queue, and redistribute the resources in the message queue when unbalanced load occurs.
The processor referred to in this application may be referred to as a CPU (Central Processing Unit ), possibly an integrated circuit chip, or a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.
The storage medium used in the present application includes various media that can store program codes, such as a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), or an optical disk.
The foregoing description is only of embodiments of the present application, and is not intended to limit the scope of the patent application, and all equivalent structures or equivalent processes using the descriptions and the contents of the present application or other related technical fields are included in the scope of the patent application.

Claims (10)

1. A method of data processing of a message queue cluster, the message queue cluster comprising a plurality of message queues of different types, the method comprising:
responding to a configuration change notification sent by a message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service;
and controlling the message queue data source service to carry out resource configuration on the message queues corresponding to the configuration information so as to load balance the message queues in the message queue cluster.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the control message queue data source service performs resource configuration on the message queue corresponding to the configuration information, and the method comprises the following steps:
acquiring a corresponding server identifier in the configuration information;
determining a target message queue from a plurality of message queues according to the server identification;
and the control message queue data source service performs resource configuration on the target message queue according to the configuration information.
3. The method of claim 2, wherein the resource configuration comprises one of deleting a resource, adding a resource, and modifying a resource.
4. The method of claim 2, wherein the configuration information includes a theme and a subtopic;
the control message queue data source service performs resource configuration on the target message queue according to the configuration information, and comprises the following steps:
the control message queue data source service determines a target data source corresponding to the target message queue;
and establishing a corresponding relation between the target data source and the topics and the subtopics.
5. The method of claim 1, wherein the configuration change notification is a new message queue notification, and wherein the obtaining corresponding configuration information from the message queue configuration service in response to receiving the configuration change notification sent by the message queue configuration service comprises:
responding to a new message queue notification sent by a message queue configuration service, and acquiring corresponding configuration information from the message queue configuration service;
the control message queue data source service performs resource configuration on the message queue corresponding to the configuration information, and the method comprises the following steps:
and the control message queue data source service adds a message queue according to the configuration information and performs resource configuration on the newly added message queue.
6. The method of claim 1, wherein the configuration information includes a message queue type, a topic, a sub-topic, and a server identification.
7. The method of claim 1, wherein prior to receiving the configuration change notification sent by the message queue configuration service, comprising:
and monitoring that the configuration information in the message queue configuration service is changed, and generating the configuration change notification.
8. The method of claim 7, wherein the monitoring for a change in configuration information in a message queue configuration service, generating the configuration change notification, comprises:
the configuration information in the message queue configuration service is monitored to be changed at regular time, and a configuration change notification is generated; or alternatively, the first and second heat exchangers may be,
and monitoring that the configuration information in the message queue configuration service is changed in real time, and generating the configuration change notification.
9. An electronic device comprising a memory for storing a computer program and a processor for executing the computer program to implement the method of any of claims 1-8.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program for implementing the method according to any of claims 1-8 when being executed by a processor.
CN202211553290.3A 2022-12-02 2022-12-02 Data processing method of message queue cluster, electronic equipment and storage medium Pending CN116319591A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211553290.3A CN116319591A (en) 2022-12-02 2022-12-02 Data processing method of message queue cluster, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211553290.3A CN116319591A (en) 2022-12-02 2022-12-02 Data processing method of message queue cluster, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116319591A true CN116319591A (en) 2023-06-23

Family

ID=86782216

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211553290.3A Pending CN116319591A (en) 2022-12-02 2022-12-02 Data processing method of message queue cluster, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116319591A (en)

Similar Documents

Publication Publication Date Title
EP3221795B1 (en) Service addressing in distributed environment
CA2644124C (en) Managing rich presence collections
KR100324976B1 (en) Data processing apparatus, method and computer program product for carrying out workload management with respect to a group of servers in an asynchronous client/server computing system
US5781737A (en) System for processing requests for notice of events
CN109660607A (en) A kind of service request distribution method, method of reseptance, device and server cluster
US20110055368A1 (en) Connection Pool Use of Runtime Load Balancing Service Performance Advisories
US20110264748A1 (en) On-demand mailbox synchronization and migration system
US7467388B2 (en) Monitoring message queues and starting processing applications
US20070239866A1 (en) Managing Rich Presence Collections
WO2018108001A1 (en) System and method to handle events using historical data in serverless systems
US7870425B2 (en) De-centralized nodal failover handling
US8984530B2 (en) Queued message dispatch
CN108696581B (en) Distributed information caching method and device, computer equipment and storage medium
US20080310439A1 (en) Communicating prioritized messages to a destination queue from multiple source queues using source-queue-specific priority values
US10530726B2 (en) Email notifications
CN109167819B (en) Data synchronization system, method, device and storage medium
US5768523A (en) Program product for processing requests for notice of events
CN110928704B (en) Message processing method, message processing system, server and computer storage medium
EP3672203A1 (en) Distribution method for distributed data computing, device, server and storage medium
CN111352746B (en) Message flow limiting method and storage medium
EP2472416A1 (en) Data query system and constructing method thereof and corresponding data query method
JP2001202318A (en) Data distribution system
CN116319591A (en) Data processing method of message queue cluster, electronic equipment and storage medium
CN111427674A (en) Micro-service management method, device and system
CN104618414A (en) Implementation method of distributed service, service agent device and distributed system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination