CN111045837A

CN111045837A - Cross-service consumption method and storage medium

Info

Publication number: CN111045837A
Application number: CN201911173038.8A
Authority: CN
Inventors: 刘德建; 林伟; 郭玉湖; 陈宏�
Original assignee: Fujian Tianquan Educational Technology Ltd
Current assignee: Fujian Tianquan Educational Technology Ltd
Priority date: 2019-11-26
Filing date: 2019-11-26
Publication date: 2020-04-21
Anticipated expiration: 2039-11-26
Also published as: CN111045837B

Abstract

The invention provides a cross-service consumption method and a storage medium, wherein the method comprises the following steps: monitoring the MQ queues in all IDCs in real time by a monitoring program according to a preset monitoring period; if monitoring that the message of the MQ queue in one IDC cannot be consumed within a preset time length, sending a stacking message corresponding to the MQ queue in the IDC to other IDCs, wherein the stacking message comprises queue connection information, a message type to be consumed and a queue name; and after receiving the stacking message, other IDCs establish a consuming thread connected to an MQ queue in the IDC according to the stacking message if local data which can be consumed within a preset time length is acquired by a monitoring program. The invention can not only realize the load balance of each cluster under the condition of not influencing the self ability, but also realize the load balance only based on the deployed service, thereby saving the cost and the resources; furthermore, manual intervention is not needed, and the intelligent control system is more intelligent.

Description

Cross-service consumption method and storage medium

Technical Field

The invention relates to the field of message middleware, in particular to a cross-service consumption method and a storage medium.

Background

Message middleware is applicable to distributed environments where reliable data transfer is required. In the system adopting the message middleware mechanism, different objects activate the event of the other side by transmitting messages, and the corresponding operation is completed. Specifically, the sender sends the message to a message server, which places the message in a number of queues and forwards the message to the recipient when appropriate. Message middleware, which is often used to mask features between various platforms and protocols, enables collaboration between applications, has the advantage of providing synchronous and asynchronous connections between clients and servers, and can deliver or store-and-forward messages at any time, which is a further reason than remote procedure calls.

In the current MQ usage scenario, a scenario in which the service of the business is suddenly increased is very often encountered. The number of messages in MQ resulting from this scenario increases substantially in a short time, which will eventually result in a large accumulation of messages in MQ due to limited queue consumption capabilities. This is due to normal traffic operation with the newly increased message rate being greater than or equal to the consumption rate. When the situation is met, message accumulation is generally discovered by the self, or the message accumulation is discovered and alarmed through specific monitoring software, finally, a consumption instance is manually and newly added manually, and consumption of the accumulated messages in the MQ is performed by an increased consumption thread. One of the above functions is not intelligent enough and requires manual handling of the accumulation of messages; meanwhile, if the newly added instance is only used for processing the accumulated messages at this time, whether the messages are deleted later or not is judged, and if the same problem needs to be processed again next time after deletion, reloading is needed; the deletion is not performed, and the idle waste of resources can be caused.

Therefore, there is a need to provide an efficient solution to the problem of MQ queue message accumulation due to the rapid increase in business services.

Disclosure of Invention

The technical problem to be solved by the invention is as follows: a cross-service consumption method and a storage medium are provided, intelligent load balancing is achieved, and manual access is not needed.

In order to solve the technical problems, the invention adopts the technical scheme that:

a method of cross-service consumption, comprising:

monitoring the MQ queues in all IDCs in real time by a monitoring program according to a preset monitoring period;

if monitoring that the message of the MQ queue in one IDC cannot be consumed within a preset time length, sending a stacking message corresponding to the MQ queue in the IDC to other IDCs, wherein the stacking message comprises queue connection information, a message type to be consumed and a queue name;

and after receiving the stacking message, other IDCs establish a consuming thread connected to an MQ queue in the IDC according to the stacking message if local data which can be consumed within a preset time length is acquired by a monitoring program.

The invention provides another technical scheme as follows:

a computer-readable storage medium, having stored thereon a computer program, which, when executed by a processor, is capable of carrying out the steps involved in the above-mentioned cross-service consumption method.

The invention has the beneficial effects that: monitoring the message accumulation condition of the MQ queues of all IDCs through a monitoring program, informing other end services to perform consumption processing when judging that the self cannot consume within the preset time, and pulling the message accumulation end to assist the consumption processing through establishing connection when the other end services confirm that the self has residual power. The invention can realize reasonable utilization of resources through monitoring based on deployed services, does not need manual access for processing, and is more intelligent.

Drawings

FIG. 1 is a flowchart illustrating a cross-service consumption method according to an embodiment of the present invention;

fig. 2 is an interaction diagram of a second embodiment of the invention.

Detailed Description

In order to explain technical contents, achieved objects, and effects of the present invention in detail, the following description is made with reference to the accompanying drawings in combination with the embodiments.

The most key concept of the invention is as follows: based on deployed services, load balancing is achieved through monitoring, manual access is not needed for processing, and the method is more intelligent.

Referring to fig. 1, the present invention provides a cross-service consumption method, including:

From the above description, the beneficial effects of the present invention are: through a method for overall monitoring and supporting cross-service consumption of each end, a special monitoring program is used for monitoring various accumulated data, when conditions are met, each end is informed to consume, and each end service needs to be given surplus power to pull the information of the accumulated end in batches after the information on the machine is normally consumed in advance, and a corresponding information consumption processing function is carried out. Therefore, reasonable utilization of resources can be realized through monitoring on deployed and used services without manual intervention for processing.

Further, still include:

and if other IDCs acquire local data which cannot be consumed within a preset time length by themselves through the monitoring program, the processing corresponding to the accumulated messages is not carried out.

As can be seen from the above description, the IDCs at other ends are only capable of facilitating cross-service consumption, so as to achieve load balancing of the whole system.

Further, the monitoring program monitors the MQ queues in each IDC in real time according to a preset monitoring period, specifically:

monitoring the message quantity to be consumed of the MQ queues in each IDC in real time by a monitoring program according to a preset monitoring period;

and the monitoring program counts the average consumption rate of the MQ queues in all the IDCs in real time according to historical monitoring data.

From the above, the monitoring program will accurately grasp the consumption condition of the MQ queue in each IDC through timing monitoring.

Further, if it is monitored that the message of the MQ queue in the IDC cannot be consumed within the preset time duration, the method specifically includes:

and if the writing rate of the MQ queue in the IDC is monitored for n times continuously and is judged to be not capable of finishing consuming in the preset time length according to the average consuming speed, judging the message accumulation of the MQ queue in the IDC, wherein n is more than or equal to 2.

From the above description, it can be known that the method comprehensively judges whether the queue has the accumulation condition through at least two conditions, and has more accuracy.

Further, after receiving the stacking message, if the other IDCs obtain local data that can be consumed within a preset time period through a monitor program, a consuming thread connected to an MQ queue in the IDC is created according to the stacking message, specifically:

after receiving the accumulation message, other IDCs respectively acquire local consumption conditions from the monitoring program and respectively judge whether the IDCs meet cross-service consumption conditions according to the local consumption conditions; the local consumption condition comprises whether message accumulation exists or not and whether local data can be consumed within a preset time length or not;

if so, creating a consuming thread connected to the MQ queue in the IDC according to the stacking message.

According to the description, each end service acquires the consumption condition of the end service through accessing the monitoring program, and judges whether the accumulation end can be helped, so that the consumption capability of the end service can be ensured not to be influenced, and the relative balance of the resource distribution of the system can be realized.

Further, still include:

and the IDC timing access monitoring program is connected with the MQ queue in the IDC to acquire the consumption condition of the MQ queue in the IDC, and if the accumulated messages can be consumed within a preset time length, the connection with the MQ queue in the IDC is closed.

As can be seen from the above description, the system has the function of monitoring whether the 'rescue' can be stopped in real time, and avoids wasting resources.

The invention provides another technical scheme as follows:

a computer-readable storage medium, having stored thereon a computer program enabling, when executed by a processor, to carry out the steps comprised in a method of cross-service consumption as described above.

The steps include:

Further, still include:

It should be understood by those skilled in the art that all or part of the processes in the above technical solutions may be implemented by instructing the related hardware through a computer program, where the program may be stored in a computer-readable storage medium, and when executed, the program may include the processes of the above methods.

The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

Example one

The embodiment provides a cross-service consumption method, which comprises the following steps:

s1: and the monitoring program monitors the messages of the MQ queues in all the IDCs in real time according to a preset monitoring period.

The IDC is a computer room, represents a data center, and can be understood as a computer room, and one MQ cluster can be deployed on a plurality of servers in one computer room.

Under a normal scene, the MQ in each IDC can normally consume the corresponding message in the system of the IDC, and a large amount of data accumulation cannot be caused. In this embodiment, there is an independent monitoring program, which monitors the number of messages to be consumed by the queue in MQ of each IDC at preset monitoring time intervals, such as 15 seconds; meanwhile, according to historical monitoring data, the average message consumption rate in each MQ can be counted. Each moment MQ has its own consumption rate, and the average value in a period of time can be calculated accordingly.

S2: and if monitoring that the message of the MQ queue in one IDC cannot be consumed within a preset time length, sending a stacking message corresponding to the MQ queue in the IDC to other IDCs, wherein the stacking message comprises queue connection information, a message type to be consumed and a queue name.

Specifically, if the writing rate of the MQ queue in the IDC is monitored n times continuously and is greater than the consumption rate, and it is determined that the message cannot be consumed within the preset time duration according to the average consumption speed, it is determined that the message accumulation condition of the MQ queue in the IDC exists. N is greater than or equal to 2, preferably 4; the predetermined period of time is generally a short period of time, for example 5 minutes, preferably 2 to 10 minutes.

After the accumulation condition is judged to exist, the corresponding IDC is an accumulation end, and the corresponding IDC sends an accumulation message corresponding to the IDC to other IDCs to inform the IDCs that the IDCs are in the accumulation state at present and request help. Wherein the stacked message includes queue connection information (including domain name, port, username, and password), a type of message to be consumed, and a queue name. The pile-up message is used for other clusters to connect to the self cluster accordingly and to explicitly consume the object.

S3: and after receiving the stacking message, other IDCs establish a consuming thread connected to an MQ queue in the IDC according to the stacking message if local data which can be consumed within a preset time length is acquired by a monitoring program.

Specifically, after receiving a stacking message sent by a stacking end, other IDCs need to respectively acquire local consumption conditions from a monitoring program, and respectively judge whether the IDCs meet cross-service consumption conditions according to the local consumption conditions; the local consumption condition comprises whether message accumulation exists or not and whether local data can be consumed within a preset time length (the preset time length is the same as the preset time length) or not; if yes, the situation that the accumulated information is consumed by the accumulation end is solved, and therefore the accumulation end is helped to consume the accumulated information by creating a consumption thread connected to an MQ queue in the IDC according to the accumulated information; if not, it is proved that no residual force exists to help other terminals, so that the processing corresponding to the stacked message is not performed, and the stacked message can be regarded as the stacked message received in a direct mode.

S4: the IDC establishing connection with the MQ queue (namely the stacking end) in the IDC accesses the monitoring program regularly, acquires the consumption condition of the MQ queue (namely the stacking end) in the IDC, and closes the connection with the MQ queue in the IDC if the IDC can consume the stacked messages within a preset time length. Namely, when the accumulation end has the capacity of consuming the accumulation messages within the preset time length, the connection is closed in time, and cross-service consumption is cancelled in time, so that the reasonable utilization of double-end resources is realized.

Example two

Referring to fig. 2, the first embodiment of the present invention provides a specific application scenario:

assume that there are 3 i.e. three IDC bays: A. and B and C, each IDC machine room is provided with a set of MQ message middleware for asynchronous message consumption. Assuming that the 3 sets of MQ clusters all consume corresponding messages in their own system, the message types and message queues are the same (assuming that there is one type, one message queue in each cluster, and the types and message queue names in 3 clusters are the same), but are distributed in 3 IDCs, i.e., 3 different environments.

Under a normal use scene, the MQ in each IDC can normally consume the corresponding message in the system of the IDC, and a large amount of data accumulation cannot be caused.

There is a separate monitoring program that monitors the piled-up messages of MQ in each IDC every 15 seconds; meanwhile, according to historical monitoring data, the average message consumption rate in each MQ can be counted. The MQ has the consumption rate of the MQ at each moment, and the average value in a period of time can be counted.

Suppose at a certain moment a large accumulation of messages is generated in IDC _ B, totaling around 10W, and the message writing rate is greater than the message consumption rate. At this point, the monitor finds the message pile up in the MQ within 4 consecutive time intervals (within 1 consecutive minute), and calculates that it cannot be consumed in a short time, say 5 minutes, based on its average message consumption rate. At this time, the monitor sends message pile-up messages for IDC _ B to IDC _ a and IDC _ C clusters.

The message match transmission data comprises: IDC _ B's associated connection messages (including domain name, port, username, and password), type of consumption, and name of queue. Other clusters may be connected into IDC _ B according to these parameters; it is also known what type of message needs to be consumed and what the queue name is consumed.

When IDC _ A and IDC receive the message from the monitor, first obtain the local consumption condition to the monitor, and condition that the same message is not piled up and the local data can be consumed within 5 minutes according to the average message rate. When the condition is not met, the notification message is directly abandoned; when this condition is satisfied, a message consuming action will take place where the corresponding consumer thread is created natively, connected to IDC _ B.

Meanwhile, the IDC connected to the IDC _ B accesses the monitoring program at intervals to acquire the conditions of message accumulation, average consumption rate and the like of the cluster B, and if the conditions reach that the IDC can consume the accumulated messages, the consumption connection of the IDC is closed, and cross-cluster consumption processing is not performed.

EXAMPLE III

This embodiment corresponds to the first embodiment and the second embodiment, and provides a computer-readable storage medium, on which a computer program is stored, where the computer program is capable of implementing the steps included in the cross-service consumption method described in the first embodiment or the second embodiment when the computer program is executed by a processor. The detailed steps are not repeated here, and refer to the descriptions of the first embodiment and the second embodiment for details.

In summary, the cross-service consumption method and the storage medium provided by the present invention can not only achieve load balancing of each cluster, but also achieve the load balancing only based on the deployed service, which saves cost and resources; furthermore, manual intervention is not needed, and the intelligent operation is realized; in addition, the consumption capability of the system can be guaranteed not to be affected, and the system has high practicability; meanwhile, the invention also has the advantage of simple and convenient implementation.

The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent changes made by using the contents of the present specification and the drawings, or applied directly or indirectly to the related technical fields, are included in the scope of the present invention.

Claims

1. A method of cross-service consumption, comprising:

2. A method of cross-service consumption as recited in claim 1, further comprising:

3. The method for cross-service consumption as claimed in claim 1, wherein the monitoring program monitors the MQ queues in each IDC in real time according to a preset monitoring period, specifically:

4. The method of claim 1, wherein if the message monitoring the MQ queue in the IDC fails to be consumed within a predetermined time period, the method further comprises:

5. The method according to claim 1, wherein after receiving the stacked message, the other IDCs create a consuming thread connected to the MQ queue in the IDC according to the stacked message if local data that can be consumed within a preset time period is obtained through a monitor, specifically:

6. A method of cross-service consumption as recited in claim 1, further comprising:

7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, is adapted to carry out the steps of a cross-service consumption method according to any one of the preceding claims 1 to 6.