CN113055483A - Message middleware data processing method, device and system - Google Patents
Message middleware data processing method, device and system Download PDFInfo
- Publication number
- CN113055483A CN113055483A CN202110295091.6A CN202110295091A CN113055483A CN 113055483 A CN113055483 A CN 113055483A CN 202110295091 A CN202110295091 A CN 202110295091A CN 113055483 A CN113055483 A CN 113055483A
- Authority
- CN
- China
- Prior art keywords
- message
- middleware
- data processing
- load balancing
- message middleware
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 33
- 238000004891 communication Methods 0.000 claims abstract description 46
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012544 monitoring process Methods 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 12
- 230000008859 change Effects 0.000 claims description 10
- 238000003825 pressing Methods 0.000 claims description 10
- 230000003247 decreasing effect Effects 0.000 claims description 7
- 238000004519 manufacturing process Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 13
- 230000008569 process Effects 0.000 description 13
- 239000000872 buffer Substances 0.000 description 6
- 230000009467 reduction Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 1
- 230000009172 bursting Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the application provides a message middleware data processing method, a device and a system, which can also be used in the financial field, wherein the method comprises the following steps: receiving a message sent by a soft load balancing system, wherein the message is sent after a message producer establishes a communication short connection with the soft load balancing system; executing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the received message within a set time period, establishing communication long connection with a set message middleware after the capacity adjustment operation is completed, and forwarding the message to the message middleware; the method and the device can effectively reduce the connection number and the operation pressure of the message middleware and ensure the working performance of the message middleware.
Description
Technical Field
The application relates to the field of data processing, can also be used in the field of finance, and particularly relates to a message middleware data processing method, device and system.
Background
Prior art message middleware, such as Kafka, is used to relay messages, where message producers push messages to Kafka, and consumers of messages subscribe to data from Kafka. By introducing kafka, the IT system can be decoupled, and asynchronous operation is realized in a message mode.
The inventors have found that kafka establishes a long connection with each message producer in actual use to quickly receive messages. As the number of producer connections continues to increase, the pressure on kafka increases, and on a certain scale, the performance of kafka is significantly affected.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a message middleware data processing method, device and system, which can effectively reduce the connection number and the operation pressure of the message middleware and ensure the working performance of the message middleware.
In order to solve at least one of the above problems, the present application provides the following technical solutions:
in a first aspect, the present application provides a message middleware data processing method, including:
receiving a message sent by a soft load balancing system, wherein the message is sent after a message producer establishes a communication short connection with the soft load balancing system;
and executing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the received message within a set time period, establishing communication long connection with a set message middleware after the capacity adjustment operation is finished, and forwarding the message to the message middleware.
Further, the executing, according to the number of message producers corresponding to the message received within the set time period, a corresponding gateway cluster capacity adjustment operation includes:
monitoring the number of message producers corresponding to the received message within a set time period;
and increasing or decreasing the corresponding number of the elastic gateways according to the numerical comparison relationship between the number of the message producers and the preset threshold value.
Further, after increasing or decreasing a corresponding number of resilient gateways according to a numerical comparison relationship between the number of message producers and a preset threshold, the method includes:
and sending the change quantity of the elastic gateways in the gateway cluster to the soft load balancing system so that the soft load balancing system updates a load balancing strategy according to the change quantity of the elastic gateways.
Further, said forwarding the message to the message middleware comprises:
pressing the message into a preset message task queue;
and monitoring whether the volume of the preset message task queue exceeds a preset threshold value, if so, sending all messages in the preset message task queue to the message middleware at one time, and if not, continuously pressing the newly received messages into the preset message task queue.
In a second aspect, the present application provides a message middleware data processing apparatus, including:
the system comprises a message receiving module, a message sending module and a message sending module, wherein the message receiving module is used for receiving a message sent by a soft load balancing system, and the message is sent after a message producer establishes communication short connection with the soft load balancing system;
and the gateway adjusting module is used for executing corresponding gateway cluster capacity adjusting operation according to the number of message producers corresponding to the received messages in a set time period, establishing communication long connection with a set message middleware after the capacity adjusting operation is finished, and forwarding the messages to the message middleware.
Further, the gateway adjusting module includes:
the message production quantity monitoring unit is used for monitoring the quantity of message producers corresponding to the received message within a set time period;
and the elastic gateway quantity adjusting unit is used for increasing or decreasing the corresponding quantity of elastic gateways according to the numerical comparison relationship between the message producer quantity and a preset threshold value.
Further, the gateway adjusting module includes:
the task queue unit is used for pressing the message into a preset message task queue;
and the queue task processing unit is used for monitoring whether the volume of the preset message task queue exceeds a preset threshold value, if so, sending all messages in the preset message task queue to the message middleware at one time, and if not, continuously pressing the newly received messages into the preset message task queue.
In a third aspect, the present application provides a message middleware data processing system, comprising: the system comprises a message producer, a soft load balancing system, a gateway cluster and message middleware;
the gateway cluster includes:
a message receiving module, configured to receive a message sent by the soft load balancing system, where the message is sent after a short communication connection is established between the message producer and the soft load balancing system;
and the gateway adjusting module is used for executing corresponding gateway cluster capacity adjusting operation according to the number of message producers corresponding to the received messages within a set time period, establishing communication long connection with the message middleware after the capacity adjusting operation is finished, and forwarding the messages to the message middleware.
In a fourth aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the message middleware data processing method when executing the program.
In a fifth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the message middleware data processing method.
According to the technical scheme, the message middleware data processing method, the message middleware data processing device and the message middleware data processing system are characterized in that communication short connection is established with a message producer through a soft load balancing system, the message sent by the message producer is received, the message is sent to a gateway cluster after being subjected to load balancing through the soft load balancing system, meanwhile, the gateway cluster executes corresponding gateway cluster capacity adjustment operation according to the number of the message producers corresponding to the message received within a set time period, and then forwards the message to the corresponding message middleware, so that the connection number and the operation pressure of the message middleware are effectively reduced, and the working performance of the message middleware is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart illustrating a data processing method of message middleware according to an embodiment of the present application;
FIG. 2 is a second flowchart illustrating a data processing method of the message middleware according to an embodiment of the present application;
fig. 3 is a third flowchart illustrating a data processing method of the message middleware according to an embodiment of the present application;
FIG. 4 is one of the block diagrams of a message middleware data processing apparatus in the embodiment of the present application;
FIG. 5 is a second block diagram of a message middleware data processing apparatus according to an embodiment of the present application;
FIG. 6 is a third block diagram of a message middleware data processing apparatus according to an embodiment of the present application;
FIG. 7 is a block diagram of a message middleware data processing system in an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It is considered that in the prior art, the message middleware establishes a long connection with each message producer in actual use so as to quickly receive the message. The application provides a method, a device and a system for processing message middleware data, wherein a communication short connection is established with a message producer through a soft load balancing system, the message sent by the message producer is received, the soft load balancing system carries out load balancing on the message and then sends the message to a gateway cluster, and meanwhile, the gateway cluster executes corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period and then forwards the message to the corresponding message middleware, so that the connection number and the operating pressure of the message middleware are effectively reduced, and the working performance of the message middleware is guaranteed.
In order to effectively reduce the connection number and the operating pressure of the message middleware and ensure the working performance of the message middleware, the application provides an embodiment of a message middleware data processing method, and referring to fig. 1, the message middleware data processing method specifically includes the following contents:
step S101: receiving a message sent by a soft load balancing system, wherein the message is sent after a message producer establishes a communication short connection with the soft load balancing system.
Optionally, the execution subject of the method of the present application may be a gateway cluster arranged between the message producer and the message middleware, for example, a cluster composed of a plurality of flexible gateways, and the cluster capacity expansion or capacity reduction may be dynamically performed according to actual performance needs, for example, when the message producer is continuously increased, the gateway needs to be increased, and it is ensured that each gateway can efficiently process the message, for example, the gateway may use a high-performance web container integrity based on the NIO to process the request.
Optionally, a soft load balancing system, i.e. an SLB, may be further disposed between the gateway cluster and the message producer, and the soft load balancing system is used for randomly loading the message to the downstream, i.e. the gateway cluster.
Specifically, when sending a message, a message producer may first establish a short communication connection with the soft load balancing system, and then send the message to the soft load balancing system, and load the message to the gateway cluster at random via the soft load balancing system.
It can be understood that in an actual application scenario, there is often a situation where a plurality of different message producers need to send messages to the same message middleware, and through the above contents of the present application, a plurality of different message producers can finally gather their messages to the same or a designated gateway cluster, and then the gateway cluster sends the messages to the target message middleware, thereby greatly reducing the number of connections of the message middleware and improving the message processing efficiency, for example, one message middleware kafka to twenty thousand message producers originally. Now 100 gateway clusters are established, each gateway cluster corresponding to 200 message producers, and then kafka only needs to correspond to these 100 gateway clusters.
Step S102: and executing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the received message within a set time period, establishing communication long connection with a set message middleware after the capacity adjustment operation is finished, and forwarding the message to the message middleware.
Optionally, the gateway cluster of the present application may be formed by combining a plurality of elastic gateways, so that cluster capacity expansion or capacity reduction may be dynamically performed according to actual performance requirements, for example, when message producers continuously increase, the number of gateways needs to be increased, and it is ensured that each gateway can efficiently process messages.
Specifically, the gateway cluster of the present application may execute a corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period, for example, if messages sent by more than ten thousand message producers are received within 1 minute, then execute a gateway cluster capacity expansion operation, and ensure that each gateway can efficiently process the message.
Optionally, after the gateway cluster dynamically adjusts the capacity, the gateway cluster may establish a long communication connection with a set message middleware and forward the message to the message middleware.
As can be seen from the above description, the message middleware data processing method provided in the embodiment of the present application can establish a short communication connection with a message producer through the soft load balancing system, receive a message sent by the message producer, perform load balancing on the message through the soft load balancing system, and send the message to the gateway cluster, and meanwhile, the gateway cluster executes a corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period, and then forwards the message to the corresponding message middleware, so as to effectively reduce the connection number and the operating pressure of the message middleware, and ensure the working performance of the message middleware.
In order to improve the message processing efficiency of the gateway cluster, in an embodiment of the message middleware data processing method of the present application, referring to fig. 2, the step S102 may further specifically include the following steps:
step S201: and monitoring the number of message producers corresponding to the received message in a set time period.
Step S202: and increasing or decreasing the corresponding number of the elastic gateways according to the numerical comparison relationship between the number of the message producers and the preset threshold value.
Optionally, the gateway cluster of the present application may execute a corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period, for example, if messages sent by more than ten thousand message producers are received within 1 minute, a gateway cluster capacity expansion operation is executed at this time, and it is ensured that each gateway can efficiently process the message, or if messages sent by less than one thousand message producers are received within 1 minute, a gateway cluster capacity reduction operation is executed at this time, and it is ensured that each gateway can efficiently process the message.
In order to improve the load balancing efficiency of the soft load balancing system, in an embodiment of the message middleware data processing method of the present application, after the step S202, the following may be further included:
and sending the change quantity of the elastic gateways in the gateway cluster to the soft load balancing system so that the soft load balancing system updates a load balancing strategy according to the change quantity of the elastic gateways.
Optionally, the soft load balancing system may use nginn or Haproxy, and the main purpose is to load the message to the downstream, i.e. gateway cluster, and the capacity increase or decrease of the gateway cluster may also make the soft load balancing system sense, for example, if nginn is used, Lua script may be used to receive the change and update the server configuration of nginn, and if Haproxy is used, etcd may be used to sense the gateway change and refresh the configuration of haprox.
In order to improve the message processing efficiency of the gateway cluster, in an embodiment of the message middleware data processing method of the present application, referring to fig. 3, the step S102 may further specifically include the following steps:
step S301: and pressing the message into a preset message task queue.
Step S302: and monitoring whether the volume of the preset message task queue exceeds a preset threshold value, if so, sending all messages in the preset message task queue to the message middleware at one time, and if not, continuously pressing the newly received messages into the preset message task queue.
Optionally, a message task queue for buffering may be set inside the gateway cluster of the present application, so as to prevent too much data from processing and not bursting a memory (i.e. preventing backpressure), and when a message exceeds a queue length or a size of the queue exceeds a set threshold (e.g. 1% of a machine memory size), a flush (Drainto) of a scanning thread is immediately triggered without waiting for a fixed time frequency to be cleared. In addition, if the mechanisms still cannot process the messages, the messages cannot be queued, and new messages are discarded, so that the memory of the machine is guaranteed against burst by the continuously increased messages.
In order to effectively reduce the number of connections and the operating pressure of the message middleware and ensure the working performance of the message middleware, the present application provides an embodiment of a message middleware data processing apparatus for implementing all or part of the contents of the message middleware data processing method, and referring to fig. 4, the message middleware data processing apparatus specifically includes the following contents:
a message receiving module 10, configured to receive a message sent by a soft load balancing system, where the message is sent after a message producer establishes a short communication connection with the soft load balancing system.
The gateway adjusting module 20 is configured to execute a corresponding gateway cluster capacity adjusting operation according to the number of message producers corresponding to the message received within a set time period, establish a long communication connection with a set message middleware after the capacity adjusting operation is completed, and forward the message to the message middleware.
As can be seen from the above description, the message middleware data processing apparatus provided in the embodiment of the present application can establish a short communication connection with a message producer through the soft load balancing system and receive a message sent by the message producer, and send the message to the gateway cluster after the soft load balancing system performs load balancing on the message, and meanwhile, the gateway cluster executes a corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period, and then forwards the message to the corresponding message middleware, so that the connection number and the operating pressure of the message middleware are effectively reduced, and the working performance of the message middleware is guaranteed.
In order to improve the message processing efficiency of the gateway cluster, in an embodiment of the message middleware data processing apparatus of the present application, referring to fig. 5, the gateway adjusting module 20 includes:
the message production number monitoring unit 21 is configured to monitor the number of message producers corresponding to the received message within a set time period.
And the flexible gateway number adjusting unit 22 is configured to increase or decrease a corresponding number of flexible gateways according to a numerical comparison relationship between the number of message producers and a preset threshold.
In order to improve the load balancing efficiency of the soft load balancing system, in an embodiment of the message middleware data processing apparatus of the present application, the following contents are further specifically included:
and the policy updating unit is used for sending the change quantity of the elastic gateways in the gateway cluster to the soft load balancing system so that the soft load balancing system updates the balancing load policy according to the change quantity of the elastic gateways.
In order to improve the message processing efficiency of the gateway cluster, in an embodiment of the message middleware data processing apparatus of the present application, referring to fig. 6, the gateway adjusting module 20 includes:
and the task queue unit 23 is configured to push the message into a preset message task queue.
And the queue task processing unit 24 is configured to monitor whether the size of the preset message task queue exceeds a preset threshold, if so, send all messages in the preset message task queue to the message middleware at one time, and otherwise, continuously press a newly received message into the preset message task queue.
To further illustrate the present invention, the present application further provides a specific application example of a system for implementing a message middleware data processing method by using the message middleware data processing apparatus, and referring to fig. 7, the specific application example specifically includes: message producer, soft load balancing system, gateway cluster, and message middleware.
The gateway cluster includes:
a message receiving module 10, configured to receive a message sent by the soft load balancing system, where the message is sent after the message producer establishes a short communication connection with the soft load balancing system.
The gateway adjusting module 20 is configured to execute a corresponding gateway cluster capacity adjusting operation according to the number of message producers corresponding to the message received within a set time period, establish a long communication connection with the message middleware after the capacity adjusting operation is completed, and forward the message to the message middleware.
Optionally, the message producer may be expressed based on a Java technology stack, where the message producer is a Java program, and maintains a thread-safe message queue (LinkBlockingQueue) inside, and writes the message to be reported into the queue first. Another thread scans the queue periodically and empties the messages in the queue once (draint). The message is then deserialized from the object into a byte stream (json format) and sent to the SLB via the Java native http client.
Optionally, when the message exceeds the length of the queue, or the size of the queue exceeds a set threshold (for example, 1% of the size of the machine memory), a flush (Drainto) of the scan thread is triggered immediately, and the flush does not wait until a fixed time frequency. In addition, if the mechanisms still cannot process the messages, the messages cannot be queued, and new messages are discarded, so that the memory of the machine is guaranteed against burst by the continuously increased messages.
The soft load balancing may use either Nginx or Haproxy, all the same, the main purpose being to load the requests randomly downstream, i.e. the resilient gateway. Note that since it is an elastic gateway, here gateway cluster addition or subtraction needs to be perceived by the SLB. If Nginx is used, Lua scripts can be used to receive such changes and update the server configuration for Nginx. In the case of Haproxy, etcd can be used to sense gateway changes and refresh the Haproxy configuration.
Optionally, in order to further improve the transmission efficiency of the http protocol, a configuration supporting http2.0 (usually, http 1.1 if not configured) may be configured on the SLB, so that the request is processed more efficiently.
The gateway cluster is used for receiving the web request, and capacity expansion or capacity reduction can be dynamically carried out according to actual performance requirements. When message producers are increased continuously, the number of gateways needs to be increased, and each gateway can process messages efficiently. The gateway employs a high performance NIO-based web container undersow to process the request.
Optionally, there is a buffer queue inside the gateway to prevent too much data processing from exploding the memory (i.e. preventing back pressure). The queue will buffer the received message for a while, similar to the producer, another thread will scan the queue periodically, emptying the queue once (draint). And then sent to kafka.
Optionally, similar to the message producer, when the message exceeds the queue length, or the size of the queue exceeds a set threshold (for example, 1% of the machine memory size), a flush (Drainto) of the scan thread is triggered immediately, and does not wait until a fixed time frequency. In addition, if the mechanisms still cannot process the messages, the messages cannot be queued, and new messages are discarded, so that the memory of the machine is guaranteed against burst by the continuously increased messages.
The message middleware is, for example, kafka, which, as seen from the kafka side, originally needs to face thousands of instances (i.e., message producers), now with intermediate gateway proxies, dramatically cuts down the number of connections. Such as originally one kafka for 2w producers. Now 100 gateways are established, each gateway for 200 producers, that kafka only needs to be for these 100 gateways.
As can be seen from the above, the present application can also achieve the following technical effects:
1. the whole process is synchronous except for two points of sending an http request and sending a kafka request, and all the other points are decoupled in a queue mode, so that the performance is strong.
2. By limiting the length and the size of the queue, the memory safety of a producer and a gateway is effectively ensured.
3. The kafka connection number is effectively reduced, and the gateway can elastically expand and has no performance problem.
In order to effectively reduce the connection number and the operating pressure of the message middleware and guarantee the working performance of the message middleware on a hardware level, the application provides an embodiment of an electronic device for implementing all or part of contents in the message middleware data processing method, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between the message middleware data processing device and relevant equipment such as a core service system, a user terminal, a relevant database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the logic controller may refer to an embodiment of the message middleware data processing method and an embodiment of the message middleware data processing apparatus in the embodiment for implementation, and the contents thereof are incorporated herein, and repeated details are not repeated.
It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), an in-vehicle device, a smart wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In practical applications, part of the message middleware data processing method may be executed on the electronic device side as described above, or all operations may be completed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.
The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
Fig. 8 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 8, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this FIG. 8 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the message middleware data processing method functions may be integrated into the central processor 9100. The central processor 9100 may be configured to control as follows:
step S101: receiving a message sent by a soft load balancing system, wherein the message is sent after a message producer establishes a communication short connection with the soft load balancing system.
Step S102: and executing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the received message within a set time period, establishing communication long connection with a set message middleware after the capacity adjustment operation is finished, and forwarding the message to the message middleware.
As can be seen from the above description, in the electronic device provided in the embodiment of the present application, a short communication connection is established with a message producer through a soft load balancing system, and the message sent by the message producer is received, and the message is sent to a gateway cluster after being load balanced by the soft load balancing system, and meanwhile, the gateway cluster executes a corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period, and then forwards the message to a corresponding message middleware, so that the connection number and the operating pressure of the message middleware are effectively reduced, and the working performance of the message middleware is ensured.
In another embodiment, the message middleware data processing apparatus may be configured separately from the central processor 9100, for example, the message middleware data processing apparatus may be configured as a chip connected to the central processor 9100, and the message middleware data processing method function is realized by the control of the central processor.
As shown in fig. 8, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 8; further, the electronic device 9600 may further include components not shown in fig. 8, which may be referred to in the art.
As shown in fig. 8, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
An embodiment of the present application further provides a computer-readable storage medium capable of implementing all steps in the message middleware data processing method whose execution subject is the server or the client in the foregoing embodiment, where the computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the computer program implements all steps in the message middleware data processing method whose execution subject is the server or the client in the foregoing embodiment, for example, when the processor executes the computer program, the processor implements the following steps:
step S101: receiving a message sent by a soft load balancing system, wherein the message is sent after a message producer establishes a communication short connection with the soft load balancing system.
Step S102: and executing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the received message within a set time period, establishing communication long connection with a set message middleware after the capacity adjustment operation is finished, and forwarding the message to the message middleware.
As can be seen from the above description, in the computer-readable storage medium provided in this embodiment of the present application, a short communication connection is established with a message producer through a soft load balancing system, and a message sent by the message producer is received, and the message is sent to a gateway cluster after being load balanced by the soft load balancing system, and meanwhile, the gateway cluster executes a corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period, and then forwards the message to a corresponding message middleware, so that the connection number and the operating pressure of the message middleware are effectively reduced, and the working performance of the message middleware is guaranteed.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A message middleware data processing method, the method comprising:
receiving a message sent by a soft load balancing system, wherein the message is sent after a message producer establishes a communication short connection with the soft load balancing system;
and executing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the received message within a set time period, establishing communication long connection with a set message middleware after the capacity adjustment operation is finished, and forwarding the message to the message middleware.
2. The message middleware data processing method of claim 1, wherein the performing corresponding gateway cluster capacity adjustment operation according to the number of message producers corresponding to the message received within a set time period comprises:
monitoring the number of message producers corresponding to the received message within a set time period;
and increasing or decreasing the corresponding number of the elastic gateways according to the numerical comparison relationship between the number of the message producers and the preset threshold value.
3. The message middleware data processing method of claim 2, after increasing or decreasing a corresponding number of resilient gateways according to a numerical comparison relationship between the number of message producers and a preset threshold, comprising:
and sending the change quantity of the elastic gateways in the gateway cluster to the soft load balancing system so that the soft load balancing system updates a load balancing strategy according to the change quantity of the elastic gateways.
4. The message middleware data processing method of claim 1 wherein forwarding the message to the message middleware comprises:
pressing the message into a preset message task queue;
and monitoring whether the volume of the preset message task queue exceeds a preset threshold value, if so, sending all messages in the preset message task queue to the message middleware at one time, and if not, continuously pressing the newly received messages into the preset message task queue.
5. A message middleware data processing apparatus, comprising:
the system comprises a message receiving module, a message sending module and a message sending module, wherein the message receiving module is used for receiving a message sent by a soft load balancing system, and the message is sent after a message producer establishes communication short connection with the soft load balancing system;
and the gateway adjusting module is used for executing corresponding gateway cluster capacity adjusting operation according to the number of message producers corresponding to the received messages in a set time period, establishing communication long connection with a set message middleware after the capacity adjusting operation is finished, and forwarding the messages to the message middleware.
6. The message middleware data processing apparatus of claim 5 wherein the gateway adjustment module comprises:
the message production quantity monitoring unit is used for monitoring the quantity of message producers corresponding to the received message within a set time period;
and the elastic gateway quantity adjusting unit is used for increasing or decreasing the corresponding quantity of elastic gateways according to the numerical comparison relationship between the message producer quantity and a preset threshold value.
7. The message middleware data processing apparatus of claim 5 wherein the gateway adjustment module comprises:
the task queue unit is used for pressing the message into a preset message task queue;
and the queue task processing unit is used for monitoring whether the volume of the preset message task queue exceeds a preset threshold value, if so, sending all messages in the preset message task queue to the message middleware at one time, and if not, continuously pressing the newly received messages into the preset message task queue.
8. A message middleware data processing system is characterized by comprising a message producer, a soft load balancing system, a gateway cluster and message middleware;
the gateway cluster includes:
a message receiving module, configured to receive a message sent by the soft load balancing system, where the message is sent after a short communication connection is established between the message producer and the soft load balancing system;
and the gateway adjusting module is used for executing corresponding gateway cluster capacity adjusting operation according to the number of message producers corresponding to the received messages within a set time period, establishing communication long connection with the message middleware after the capacity adjusting operation is finished, and forwarding the messages to the message middleware.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the message middleware data processing method according to any one of claims 1 to 4 are implemented when the processor executes the program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the message middleware data processing method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110295091.6A CN113055483A (en) | 2021-03-19 | 2021-03-19 | Message middleware data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110295091.6A CN113055483A (en) | 2021-03-19 | 2021-03-19 | Message middleware data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113055483A true CN113055483A (en) | 2021-06-29 |
Family
ID=76513593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110295091.6A Pending CN113055483A (en) | 2021-03-19 | 2021-03-19 | Message middleware data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113055483A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037753A (en) * | 2022-06-22 | 2022-09-09 | 平安科技（深圳）有限公司 | Message notification method and system |
| CN115378831A (en) * | 2022-08-19 | 2022-11-22 | 中国建设银行股份有限公司 | Monitoring method and device for message middleware server |
| CN116016662A (en) * | 2022-12-26 | 2023-04-25 | 拉扎斯网络科技（上海）有限公司 | Message sending system, method and message processing method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110239226A1 (en) * | 2010-03-23 | 2011-09-29 | Cesare Placanica | Controlling congestion in message-oriented middleware |
| CN105872080A (en) * | 2016-05-16 | 2016-08-17 | 浪潮(北京)电子信息产业有限公司 | InCloud load balancing control method and device |
| CN106210119A (en) * | 2016-07-29 | 2016-12-07 | 深圳前海微众银行股份有限公司 | Smooth capacity expansion method and system for message middleware |
| CN106487902A (en) * | 2016-10-19 | 2017-03-08 | 华迪计算机集团有限公司 | A kind of method of data capture based on message-oriented middleware and system |
| CN107145393A (en) * | 2017-04-27 | 2017-09-08 | 努比亚技术有限公司 | A kind of load regulation method, equipment and computer-readable recording medium |
| CN108681489A (en) * | 2018-05-25 | 2018-10-19 | 西安交通大学 | It is a kind of it is super calculate environment under mass data in real time acquisition and processing method |
| CN110971430A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Automatic capacity expansion and reduction control method and device, storage medium and processor |
-
2021
- 2021-03-19 CN CN202110295091.6A patent/CN113055483A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110239226A1 (en) * | 2010-03-23 | 2011-09-29 | Cesare Placanica | Controlling congestion in message-oriented middleware |
| CN105872080A (en) * | 2016-05-16 | 2016-08-17 | 浪潮(北京)电子信息产业有限公司 | InCloud load balancing control method and device |
| CN106210119A (en) * | 2016-07-29 | 2016-12-07 | 深圳前海微众银行股份有限公司 | Smooth capacity expansion method and system for message middleware |
| CN106487902A (en) * | 2016-10-19 | 2017-03-08 | 华迪计算机集团有限公司 | A kind of method of data capture based on message-oriented middleware and system |
| CN107145393A (en) * | 2017-04-27 | 2017-09-08 | 努比亚技术有限公司 | A kind of load regulation method, equipment and computer-readable recording medium |
| CN108681489A (en) * | 2018-05-25 | 2018-10-19 | 西安交通大学 | It is a kind of it is super calculate environment under mass data in real time acquisition and processing method |
| CN110971430A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Automatic capacity expansion and reduction control method and device, storage medium and processor |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037753A (en) * | 2022-06-22 | 2022-09-09 | 平安科技（深圳）有限公司 | Message notification method and system |
| CN115037753B (en) * | 2022-06-22 | 2023-10-27 | 平安科技（深圳）有限公司 | Message notification method and system |
| CN115378831A (en) * | 2022-08-19 | 2022-11-22 | 中国建设银行股份有限公司 | Monitoring method and device for message middleware server |
| CN116016662A (en) * | 2022-12-26 | 2023-04-25 | 拉扎斯网络科技（上海）有限公司 | Message sending system, method and message processing method |
| CN116016662B (en) * | 2022-12-26 | 2024-09-17 | 拉扎斯网络科技（上海）有限公司 | Message sending system, method and message processing method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113055483A (en) | Message middleware data processing method, device and system | |
| CN111031058A (en) | Websocket-based distributed server cluster interaction method and device | |
| AU2019256257B2 (en) | Processor core scheduling method and apparatus, terminal, and storage medium | |
| CN110166791B (en) | Connection establishing method, device, equipment and storage medium | |
| CN110764881A (en) | Distributed system background retry method and device | |
| CN113055479A (en) | Self-adaptive processing method, device and system for distributed service cluster load | |
| CN111445331A (en) | Transaction matching method and device | |
| CN112953908A (en) | Network isolation configuration method, device and system | |
| CN113032233A (en) | Distributed service cluster runtime parameter adaptive processing method, device and system | |
| CN111338905A (en) | Application node data processing method and device | |
| CN112689012A (en) | Cross-network proxy communication method and device | |
| CN111240858A (en) | Event scheduling method and component | |
| CN111338780B (en) | Transmission method and device for concurrent files | |
| CN111464628B (en) | Multiplexing asynchronous processing system and method | |
| CN111510493A (en) | Distributed data transmission method and device | |
| CN114257532A (en) | Server side state detection method and device | |
| CN112764919A (en) | Distributed service self-adaptive load balancing method and device | |
| CN112559158A (en) | Micro-service timing task scheduling method and device | |
| CN111782366A (en) | Distributed task scheduling method and device | |
| CN116185755A (en) | Data processing method and device for distributed load balancing system | |
| CN115562898A (en) | Distributed payment system exception handling method and device | |
| CN114697339A (en) | Load balancing method and device under centralized architecture | |
| CN114710496A (en) | Multi-node load balancing method and device | |
| CN115099930A (en) | Financial business data processing method and device | |
| CN114416212A (en) | Mobile terminal application program data resource processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210629 |
|
| RJ01 | Rejection of invention patent application after publication |