CN114584566A - Data processing method, device and equipment based on message queue and storage medium - Google Patents
Data processing method, device and equipment based on message queue and storage medium Download PDFInfo
- Publication number
- CN114584566A CN114584566A CN202210151308.0A CN202210151308A CN114584566A CN 114584566 A CN114584566 A CN 114584566A CN 202210151308 A CN202210151308 A CN 202210151308A CN 114584566 A CN114584566 A CN 114584566A
- Authority
- CN
- China
- Prior art keywords
- server node
- message queue
- data processing
- data
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 57
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000001514 detection method Methods 0.000 claims abstract description 16
- 230000002159 abnormal effect Effects 0.000 claims description 22
- 238000012790 confirmation Methods 0.000 claims description 9
- 239000000523 sample Substances 0.000 claims description 6
- 229920006395 saturated elastomer Polymers 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 2
- 230000006854 communication Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 12
- 230000036541 health Effects 0.000 description 7
- 230000007175 bidirectional communication Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1029—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers using data related to the state of servers by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1031—Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention provides a data processing method, a device, equipment and a computer readable storage medium based on a message queue, wherein the method comprises the following steps: the method comprises the steps of obtaining data in a local message queue, sending a detection message to a server node according to a preset frequency after obtaining the data in the local message queue, determining the load state of the server node according to a determination request message returned by the server node, and finally distributing the data to the server node for caching through a remote node queue according to the load state. The data in the original message queue is placed in the gateway processing program for distribution processing, so that the effect of reducing the running load of the original message queue is achieved, the resource utilization rate and the system processing capacity are improved, and the problem of low data processing efficiency caused by the performance of the message queue is solved.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data method, apparatus, device, and computer-readable storage medium based on a message queue.
Background
MQ (Message Queue) is a kind of data structure in the basic data structure, first in first out. Refers to placing data (messages) to be transmitted in a queue, and using a queue mechanism to implement message delivery-that is, a producer generates a message and places the message in a queue, and then the consumer processes the message. The consumer can pull the message to the designated queue or subscribe to the corresponding queue, and the message is pushed by the MQ server.
The MQ is good at caching data, and is mainly used for ensuring that the data in transmission is not lost when downtime and other conditions occur, so that the stability and the safety of communication are improved; the performance of the MQ for processing data is not strong, and in a high concurrency service scene, the problem of overlarge operation load caused by insufficient data processing capability of the MQ easily occurs, so that the data processing is unsmooth.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide a data processing method based on a message queue, aiming at solving the problem of low data processing efficiency of the message queue.
In order to achieve the above object, the present invention provides a method for processing data based on a message queue, where the method for processing data based on a message queue includes:
acquiring data in a local message queue;
after the data in the local message queue is acquired, sending a detection message to a server node according to a preset frequency, and determining the load state of the server node according to a determination request message returned by the server node;
and distributing the data to the server node for processing through a remote node queue according to the load state.
Optionally, after the step of acquiring the data in the local message queue, the method further includes:
determining whether the load state of the server node is a normal state;
if the load state is the normal state, allowing the data to be distributed to the server nodes; otherwise, the data is not allowed to be distributed to the server node.
Optionally, the step of sending a probe packet to a server node according to a preset frequency, and determining a load state of the server node according to a determination request packet returned by the server node includes:
sending the detection message to a port of the server node according to a preset frequency;
and if the port returns the confirmation request message within the preset time, judging that the load state is the normal state.
Optionally, after the step of sending the probe packet by the port of the server node according to the preset frequency, the method further includes:
when the port does not return the confirmation request message within the preset time, judging that the load state is an abnormal state;
and when the load state is an abnormal state, deleting the server node, and calling other server nodes to replace the server node to work, wherein the other server nodes are the server nodes with the load state being the normal state.
Optionally, the step of allocating the data to the server node for processing through a remote node queue according to the load status includes:
acquiring the load state of the server node;
and distributing the data to the server nodes for processing through a remote node queue by a load balancing algorithm according to the load state.
Optionally, after the step of distributing the data to the server node for processing through a remote node queue according to the load status, the method further includes:
when the load state of the server node is in the abnormal state, determining whether the size of data in the server node exceeds a preset threshold value;
and when the data size of the server node exceeds the preset threshold value, judging that the server node is in a saturated state, and adding the server node.
In addition, to achieve the above object, the present invention further provides a data processing apparatus based on a message queue, including:
the data acquisition module is used for acquiring data in the message queue;
the data processing module is used for sending a detection message to a server node according to a preset frequency and determining the load state of the server node according to a determination request message returned by the server node;
and the data caching module is used for distributing the data to the server node for processing through a remote node queue according to the load state.
In addition, to achieve the above object, the present invention further provides a message queue based data processing device, which includes a memory, a processor and a message queue based data processing program stored in the memory and executable on the processor, wherein when the processor executes the message queue based data processing program, the message queue based data processing device implements any one of the steps of the message queue based data processing method as described in any one of the above.
Furthermore, to achieve the above object, the present invention also provides a computer readable storage medium, having a message queue based data processing program stored thereon, which when executed by a processor implements any one of the steps of the message queue based data processing method as described in any one of the above.
The embodiment of the invention provides a data processing method, a device, equipment and a computer readable storage medium based on a message queue, wherein the method comprises the following steps: the method comprises the steps of obtaining data in a local message queue, sending a detection message to a server node according to a preset frequency after obtaining the data in the local message queue, determining the load state of the server node according to a determination request message returned by the server node, and finally distributing the data to the server node for caching through a remote node queue according to the load state. The data in the original message queue is placed in the gateway processing program for distribution processing, so that the effect of reducing the running load of the original message queue is achieved, and the problem of low data processing efficiency caused by the performance of the message queue is solved.
Drawings
FIG. 1 is a schematic diagram of the overall architecture of an MQ cluster of the present invention;
FIG. 2 is a diagram illustrating a hardware architecture of a message queue-based data processing apparatus according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a first embodiment of a message queue-based data processing method according to the present invention;
FIG. 4 is a flowchart illustrating a second embodiment of a message queue-based data processing method according to the present invention;
fig. 5 is a detailed flowchart of step S20 in the third embodiment of the data processing method based on message queue according to the present invention;
FIG. 6 is a flowchart illustrating a fourth embodiment of a message queue-based data processing method according to the present invention
Fig. 7 is a detailed flowchart of step S30 in the fifth embodiment of the data processing method based on message queue according to the present invention;
FIG. 8 is a flowchart illustrating a sixth embodiment of a message queue-based data processing method according to the present invention;
FIG. 9 is a block diagram of a message queue-based data processing apparatus according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It is to be understood that the appended drawings illustrate exemplary embodiments of the invention, which may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
The IBM MQ carries out a clustering function and simultaneously realizes load balancing, and is mainly realized by using a shared queue mechanism, but in a high concurrent service scene, a performance bottleneck exists. The invention aims to provide and realize an MQ cluster scheme based on a multi-process program, the cluster routing and load balancing functions are independently processed by a multi-process gateway-like processing program, the program adopts a multi-process mode, the number of processes is synchronously increased and decreased along with the size of the traffic, the resource utilization rate and the system processing capacity are improved as much as possible, and the MQ cluster processing performance is improved. The MQ returns to the direction of good communication stability and security, and the problem of performance bottleneck of the IBM MQ self-carried cluster is solved.
Referring to fig. 1, fig. 1 is a schematic diagram of an overall architecture of an MQ cluster according to the present invention, where the MQ cluster adopts a design mode of cluster routing/load balancing and message communication separation, and consists of two parts, namely a class gateway processing program and an MQ intermediate communication device. An MQ queue manager in a class gateway processing program establishes a class gateway local receiving queue, a node 1 remote queue, a node 2 remote queue and a node 3 remote queue; the node 1 queue manager establishes a node 1 local receiving queue and a class gateway remote receiving queue; the node 2 queue manager establishes a node 2 local receiving queue and a class gateway remote receiving queue; the node 3 queue manager establishes a node 3 local receiving queue and a class gateway remote receiving queue; and realizing the bidirectional communication between the class gateway processing program and the nodes 1, 2 and 3.
The class gateway processing program is responsible for routing and load balancing of messages, and when external messages are sent to a receiving queue of the class gateway mq, the class gateway processing program forwards the messages to remote queues of three nodes in a balanced manner by adopting a random method, so that the purposes of routing and load balancing are achieved. The program adopts a multi-process mode, the number of processes is synchronously increased and decreased along with the size of the service volume, and the resource utilization rate and the system processing capacity are improved as much as possible, so that the processing performance of the MQ cluster is improved.
In addition, the similar gateway processing program sends communication detection messages to the node 1, the node 2 and the node 3 at regular time, if a certain node is down, the similar gateway processing program automatically deletes the node through a load balancing strategy, and the rest two nodes replace the down node to work so as to realize the high availability of the cluster.
When the service volume is increased and the interior of the MQ cluster needs to be expanded, only the application node server needs to be added, and the MQ bidirectional communication of the configuration and class gateway processing program is needed, so that the expansibility of the whole system is improved.
As an implementation, a message queue based data processing apparatus may be as shown in fig. 2.
The embodiment of the invention relates to a data processing device based on a message queue, which comprises: a processor 101, e.g. a CPU, a memory 102, a communication bus 103. Wherein a communication bus 103 is used for enabling the connection communication between these components.
The memory 102 may be a high-speed RAM memory or a non-volatile memory (e.g., a disk memory). As shown in fig. 2, a message queue-based data processing program may be included in the memory 102, which is a computer-readable storage medium; and the processor 101 may be configured to invoke the message queue based data handler stored in the memory 102 and perform the following operations:
acquiring data in a local message queue;
after the data in the local message queue is acquired, sending a detection message to a server node according to a preset frequency, and determining the load state of the server node according to a determination request message returned by the server node;
and distributing the data to a cache in the server node through a remote node queue according to the load state.
In one embodiment, the processor 101 may be configured to invoke a message queue based data handler stored in the memory 102 and perform the following operations:
determining whether the load state of the server node is a normal state;
if the load state is the normal state, allowing the data to be distributed to the server nodes; otherwise, the data is not allowed to be distributed to the server node.
In one embodiment, the processor 101 may be configured to invoke a message queue based data handler stored in the memory 102 and perform the following operations:
sending the detection message to a port of the server node according to a preset frequency;
and if the port returns the confirmation request message within the preset time, judging that the load state is the normal state.
In one embodiment, the processor 101 may be configured to invoke a message queue based data handler stored in the memory 102 and perform the following operations:
when the port does not return the confirmation request message within the preset time, judging that the load state is an abnormal state;
and when the load state is an abnormal state, deleting the server node, and calling other server nodes to replace the server node to work, wherein the other server nodes are the server nodes with the load state being the normal state.
In one embodiment, the processor 101 may be configured to invoke a message queue based data handler stored in the memory 102 and perform the following operations:
acquiring the load state of the server node;
and distributing the data to the cache in the server node through a remote node queue by a load balancing algorithm according to the load state.
In one embodiment, the processor 101 may be configured to invoke a message queue based data handler stored in the memory 102 and perform the following operations:
when the load state of the server node is in the abnormal state, determining whether the size of data in the server node exceeds a preset threshold value;
and when the data size of the server node exceeds the preset threshold value, judging that the server node is in a saturated state, and adding the server node.
Based on the hardware architecture of the data processing device based on the message queue based on the communication technology, the embodiment of the data processing method based on the message queue is provided.
Referring to fig. 3, in a first embodiment, the message queue-based data processing method includes the following steps:
step S10: acquiring data in a local message queue;
in this embodiment, first, the message queue is set to a communication mode (in this mode, the message queue is only used for caching data and does not perform processing), the local message queue stores data received by the local port, and in this step, acquiring the data in the local message queue means transferring the data in the queue to the class gateway processing program of this application for subsequent processing.
Step S20: after the data in the local message queue is acquired, sending a detection message to a server node according to a preset frequency, and determining the load state of the server node according to a determination request message returned by the server node;
in this embodiment, after acquiring the data in the local message queue, the class gateway processing program sends a probe packet to a server node in the MQ cluster according to a certain preset frequency, and determines a load state of the server node according to a determination request packet returned by the server node, where the load state is a quantized state that reflects an operation load of the server node, and generally, a smaller load state value means a smaller operation load of the server node, that is, a stronger data processing capability of the server node.
Step S30: and distributing the data to the server node for processing through a remote node queue according to the load state.
In this embodiment, after the load state is determined, the data of the local port is allocated to the remote server node through the pre-established remote node queue for processing, so as to implement independent processing of the cluster routing and load balancing functions by the multiprocess gateway processing program, and return the MQ to the direction of the communication stability and security which is good at.
In the technical scheme provided by the embodiment, an MQ cluster is built, and the MQ cluster comprises a class gateway processing program and an MQ intermediate communication piece, wherein the MQ intermediate communication piece is used as a connection channel and used for transmitting data, and the class gateway processing program is used for processing data. By building the MQ cluster architecture, the traditional message queue does not process the message queue any more, so that the message queue is concentrated on the data cache of the message queue, and the data processing work is transferred to the MQ cluster built in the embodiment.
Referring to fig. 4, based on the first embodiment, in the second embodiment, after the step S10, the method further includes:
step S40: determining whether the load state of the server node is a normal state;
step S50: if the load state is the normal state, allowing the data to be distributed to the server nodes; otherwise, the data is not allowed to be distributed to the server node.
Optionally, this embodiment provides a manner of determining a load state of a server node, in this embodiment, the load state is divided into a normal state and other abnormal states except the normal state, and in this embodiment, whether the server node is in the normal state is determined, and when the server node does not satisfy the normal state condition, the server node is determined to be in the abnormal state. In a normal state, the server node can normally process data sent by the class gateway handler, and otherwise cannot normally process the data.
In the technical scheme provided by this embodiment, by limiting the condition that the server node is in a normal state, the class gateway program sends data to the server node after determining whether the server node is available for distributing data, which is also used as a data processing mode, so that data loss caused by an abnormal condition that the server node cannot process data is avoided, and the data processing efficiency is improved.
Referring to fig. 5, in a third embodiment, based on the first embodiment, the step S20 includes:
step S21: sending the detection message to a port of the server node according to a preset frequency;
step S22: and if the port returns the confirmation request message within the preset time, judging that the load state is the normal state.
Optionally, the present embodiment provides a way to determine whether a server node is in a normal state, in order to monitor a healthy server, a health check is generally performed to try to connect to the server through a configured protocol and a port to ensure that the server is monitoring, and the health check is used to check the availability status of various services opened by the server. The load balancing device will typically configure various health check methods such as Ping, TCP, UDP, HTTP, FTP, DNS, etc. The load balancing device checks that the 80 port of this server is DOWN and the load balancing device will not forward the following connection to this server but will instead forward the packet to another server according to an algorithm. The interval and number of attempts of the check may be set when creating the health check, for example, the interval is set to 5 seconds, the number of attempts is set to 3, the health check is initiated by the load balancing device every 5 seconds, if the check fails, the attempt is performed 3 times, if the check fails 3 times, the service is marked as DOWN, then the server will still check the server of DOWN every 5 seconds, and when the health check of the server is found to be successful again at a certain time, the server is marked as UP again. The interval time and the trial times of the health check are set according to the comprehensive condition, and the principle is that the service is not influenced and the load balancing equipment is not greatly burdened.
In the technical scheme provided by this embodiment, the server load state is judged to be normal by sending the detection message to the server node and receiving the detection message returned by the server node within the preset time, so that the data processing efficiency is improved.
Referring to fig. 6, in the fourth embodiment, after step S21, based on the first embodiment, the method further includes:
step S60: when the port does not return the confirmation request message within the preset time, judging that the load state is an abnormal state;
step S70: and when the load state is an abnormal state, deleting the server node, and calling other server nodes to replace the server node to work, wherein the other server nodes are the server nodes with the load state being the normal state.
Optionally, this embodiment provides a processing method when the load state is in an abnormal state, in this embodiment, the determination request packet is a connection identifier for proving that a communication request is established between the server and the class gateway handler, which is returned to the class gateway handler by the server node when receiving the probe packet, and when the port of the server node does not return the determination request packet after a preset time, it is determined that the load state of the server node is in the abnormal state, in this embodiment, when the load state is in the abnormal state, the control program in the class gateway handler deletes the server node, and it needs to be emphasized that deleting the server node refers to deleting the server node from the optional node list of the class gateway handler, that is, temporarily taking the server node as an unselected node, and when the server node can return a determined request message to the class gateway processing program again within the preset time, the server node is added into the optional node list of the class gateway processing program again.
In the technical solution provided in this embodiment, when the port does not return the confirmation request packet within the preset time, it is determined that the load state is an abnormal state, and when the load state is an abnormal state, the server node is deleted, and another server node is called to replace the server node to work, thereby improving the efficiency of data processing.
Referring to fig. 7, in a fifth embodiment, based on the first embodiment, the step S30 includes:
step S31: acquiring the load state of the server node;
step S32: and distributing the data to the cache in the server node through a remote node queue by a load balancing algorithm according to the load state.
Optionally, this embodiment provides a manner of distributing data to a server node according to a load state, after the state of the server node is determined, a class gateway processing program obtains a load state value of the server node, and distributes the data to a cache in the server node through a remote node queue by using a load balancing algorithm, where the load balancing algorithm is an algorithm that is executed by uniformly distributing requests/data to a plurality of operation units according to the state of the server, and a common load balancing algorithm includes algorithms such as weighted polling, weighted minimum connection, and the like.
In the technical scheme provided by this embodiment, when an external message is sent to the mq receive queue of the class gateway processing program, the class gateway processing program uses a random method to balance and forward the message to the remote queues of the three nodes, so as to achieve the purpose of routing and load balancing, and improve the efficiency of data processing.
Referring to fig. 8, in the sixth embodiment, after step S30, the method further includes:
step S80: when the load state of the server node is in the abnormal state, determining whether the size of data in the server node exceeds a preset threshold value;
step S90: and when the data size of the server node exceeds the preset threshold value, judging that the server node is in a saturated state, and adding the server node.
Optionally, this embodiment provides a processing method when the load state of the server node is in the abnormal state. Specifically, when the load state is in the abnormal state, the class gateway processing program acquires the data size of the current server node, and when the data size exceeds a preset threshold value in the class gateway processing program, it is determined that the server node is in a saturated state, that is, the server node is in a running state with a heavy load, at this time, the MQ bidirectional communication queue of the class gateway processing program can be configured, and in this process, the MQ bidirectional communication queue is transparent to an external system, so that the expansibility of the whole system is improved.
In the technical scheme provided by this embodiment, whether the server nodes are in a saturated state is determined by setting a determination threshold, so as to expand the number of the server nodes, and avoid that the data processing efficiency of the server nodes is reduced due to an excessively high load.
In addition, referring to fig. 9, this embodiment further proposes a message queue-based data processing apparatus, which includes:
a data obtaining module 100, configured to obtain data in a message queue;
the data processing module 200 is configured to send a detection packet to a server node according to a preset frequency, and determine a load state of the server node according to a determination request packet returned by the server node;
and the data caching module 300 is configured to allocate the data to the cache in the server node through a remote node queue according to the load state.
In addition, the present invention also provides a data processing device based on a message queue, comprising: a memory, a processor and a message queue based data processing program stored on the memory and operable on the processor, the message queue based data processing program when executed by the processor implementing the steps of the message queue based data processing method as claimed in any one of the above.
Furthermore, the present invention also provides a computer-readable storage medium, which stores a message queue-based data processing program, and when the message queue-based data processing program is executed by a processor, the message queue-based data processing program implements the steps of the message queue-based data processing method according to the above embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk) as described above, and includes several instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A message queue-based data processing method, wherein the message queue-based data processing method comprises the steps of:
acquiring data in a local message queue;
after the data in the local message queue is acquired, sending a detection message to a server node according to a preset frequency, and determining the load state of the server node according to a determination request message returned by the server node;
and distributing the data to the server node for processing through a remote node queue according to the load state.
2. The message queue-based data processing method as claimed in claim 1, wherein the local message queue comprises a local message queue and a roaming message queue, and the data is locally transmitted through the local message queue and/or is roaming-transmitted through the roaming message queue.
3. The message queue-based data processing method of claim 1, wherein after the step of obtaining data in the local message queue, further comprising:
determining whether the load state of the server node is a normal state;
if the load state is the normal state, allowing the data to be distributed to the server nodes; otherwise, the data is not allowed to be distributed to the server node.
4. The message queue-based data processing method according to claim 1, wherein the step of sending a probe packet to a server node according to a preset frequency and determining a load status of the server node according to a determination request packet returned by the server node comprises:
sending the detection message to a port of the server node according to a preset frequency;
and if the port returns the confirmation request message within the preset time, judging that the load state is the normal state.
5. The message queue-based data processing method of claim 4, wherein after the step of sending the probe packet by the port of the server node according to the preset frequency, the method further comprises:
when the port does not return the confirmation request message within the preset time, judging that the load state is an abnormal state;
and when the load state is an abnormal state, deleting the server node, and calling other server nodes to replace the server node to work, wherein the other server nodes are the server nodes with the load state being the normal state.
6. The message queue-based data processing method of claim 1, wherein the step of distributing the data through a remote node queue to the server node for processing according to the load status comprises:
acquiring the load state of the server node;
and distributing the data to the server nodes for processing through a remote node queue by a load balancing algorithm according to the load state.
7. The message queue-based data processing method of claim 1, wherein after the step of distributing the data through a remote node queue for processing in the server node according to the load status, further comprising:
when the load state of the server node is in the abnormal state, determining whether the size of data in the server node exceeds a preset threshold value;
and when the data size of the server node exceeds the preset threshold value, judging that the server node is in a saturated state, and adding the server node.
8. A message queue-based data processing apparatus, the message queue-based data processing apparatus comprising:
the data acquisition module is used for acquiring data in the message queue;
the data processing module is used for sending a detection message to a server node according to a preset frequency and determining the load state of the server node according to a determination request message returned by the server node;
and the data processing module is used for distributing the data to the server node for processing through a remote node queue according to the load state.
9. A message queue-based data processing apparatus, characterized in that the message queue-based data processing apparatus comprises: memory, processor and message queue based data processing program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the message queue based data processing method according to any of claims 1-7.
10. A computer-readable storage medium, on which a message queue-based data processing program is stored, which, when executed by a processor, implements the steps of the message queue-based data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210151308.0A CN114584566A (en) | 2022-02-16 | 2022-02-16 | Data processing method, device and equipment based on message queue and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210151308.0A CN114584566A (en) | 2022-02-16 | 2022-02-16 | Data processing method, device and equipment based on message queue and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114584566A true CN114584566A (en) | 2022-06-03 |
Family
ID=81770293
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210151308.0A Pending CN114584566A (en) | 2022-02-16 | 2022-02-16 | Data processing method, device and equipment based on message queue and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114584566A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103827829A (en) * | 2011-09-30 | 2014-05-28 | 甲骨文国际公司 | System and method for providing and managing message queues for multinode applications in a middleware machine environment |
| CN104333515A (en) * | 2014-11-14 | 2015-02-04 | 中国建设银行股份有限公司 | Method and device for processing SWIFT messages |
| CN105049503A (en) * | 2015-07-07 | 2015-11-11 | 深信服网络科技（深圳）有限公司 | Server dispatching method and load balancing device |
| CN108134830A (en) * | 2017-12-20 | 2018-06-08 | 马上消费金融股份有限公司 | Load-balancing method, system, device and storage medium based on message queue |
| CN109271265A (en) * | 2018-09-29 | 2019-01-25 | 平安科技（深圳）有限公司 | Request processing method, device, equipment and storage medium based on message queue |
-
2022
- 2022-02-16 CN CN202210151308.0A patent/CN114584566A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103827829A (en) * | 2011-09-30 | 2014-05-28 | 甲骨文国际公司 | System and method for providing and managing message queues for multinode applications in a middleware machine environment |
| CN104333515A (en) * | 2014-11-14 | 2015-02-04 | 中国建设银行股份有限公司 | Method and device for processing SWIFT messages |
| CN105049503A (en) * | 2015-07-07 | 2015-11-11 | 深信服网络科技（深圳）有限公司 | Server dispatching method and load balancing device |
| CN108134830A (en) * | 2017-12-20 | 2018-06-08 | 马上消费金融股份有限公司 | Load-balancing method, system, device and storage medium based on message queue |
| CN109271265A (en) * | 2018-09-29 | 2019-01-25 | 平安科技（深圳）有限公司 | Request processing method, device, equipment and storage medium based on message queue |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108124003B (en) | Network management equipment connection processing method, device and system | |
| US7486621B2 (en) | Method and apparatus for load sharing and overload control for packet media gateways under control of a single media gateway controller | |
| US6707792B1 (en) | Overload reduction in a communication system | |
| CN107135279B (en) | Method and device for processing long connection establishment request | |
| CN110958281B (en) | Data transmission method and communication device based on Internet of things | |
| US11233694B2 (en) | Method and device for processing communication path | |
| US11689646B2 (en) | Network packet processing method and apparatus and network server | |
| CN110875887A (en) | MQTT protocol-based communication interaction method and communication interaction system | |
| CN105635083A (en) | Service processing method and service processing system based on server and client architecture | |
| US20200351215A1 (en) | Data Stream Sending Method and System, and Device | |
| CN111147573A (en) | Data transmission method and device | |
| CN111522663B (en) | Data transmission method, device and system based on distributed storage system | |
| CN111506443A (en) | Service calling method, device, equipment and storage medium | |
| CN115002016A (en) | Message processing system, method, device and storage medium | |
| CN111555965B (en) | Message pushing method and system suitable for iOS client | |
| CN114584566A (en) | Data processing method, device and equipment based on message queue and storage medium | |
| CN109831467B (en) | Data transmission method, equipment and system | |
| US11575601B2 (en) | Network device and packet processing method using same | |
| EP3038290A1 (en) | Method and device for application management | |
| CN111371573B (en) | Message interaction method and device | |
| CN114268615B (en) | Service processing method and system based on TCP connection | |
| CN107276911B (en) | Network management method and system | |
| CN110324857B (en) | Method and device for processing service quality data stream | |
| US11016818B2 (en) | Method of allocating execution resources | |
| JP2008311971A (en) | Server system, load distribution method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |