CN110968431A - Message processing method, device and equipment - Google Patents
Message processing method, device and equipment Download PDFInfo
- Publication number
- CN110968431A CN110968431A CN201811142226.XA CN201811142226A CN110968431A CN 110968431 A CN110968431 A CN 110968431A CN 201811142226 A CN201811142226 A CN 201811142226A CN 110968431 A CN110968431 A CN 110968431A
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- data queue
- target mark
- position information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application provides a message processing method, a device and equipment, wherein the method comprises the following steps: storing a message to a data queue, and determining the position information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message; storing the corresponding relation between the target mark and the position information into an operation queue; when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue; and processing the message according to the target mark. According to the technical scheme, the operations of storing, updating and deleting the message are sequentially added to the queue, the operations of reading and writing the disk are sequentially performed, the time complexity is low, the bandwidth of the disk can be fully utilized, and the extremely high throughput is realized.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a method, an apparatus, and a device for processing a message.
Background
The middleware can be suitable for a distributed environment needing reliable data transmission, and in a system adopting the middleware, different objects activate events of the other party by transmitting messages to finish corresponding operations. Specifically, the sender sends the message to the middleware, which forwards the message to the recipient. Among them, middleware can communicate between different platforms, and is often used to mask features between various platforms and protocols, and implement collaboration between applications, which has the advantages of providing synchronous and asynchronous connection between sender and receiver, and storing and forwarding messages at any time.
After the sender sends the message to the middleware, the middleware needs to perform persistence processing on the message, such as storing the message in a relational database, and then forwarding the message to the receiver. Thus, once a message is unsuccessfully forwarded to a recipient, the middleware reads the message from the relational database and re-forwards the message, and so on, until the message is successfully forwarded to the recipient, the middleware deletes the message from the relational database.
However, when the middleware performs persistence processing on the message, the message is stored in the relational database, which is expensive and generates a large amount of random IO (input/output), thereby affecting stability.
Disclosure of Invention
The application provides a message processing method, which comprises the following steps:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
The application provides a message processing method, which comprises the following steps:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
and processing the message according to the position information and the target mark.
The present application provides a message processing apparatus, the apparatus comprising:
the determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message;
the storage module is used for storing the relation between the target mark and the position information to an operation queue;
the acquisition module is used for acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue when the message in the data queue is processed;
and the processing module is used for processing the message according to the target mark.
The present application provides a message processing apparatus, the apparatus comprising:
the determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message;
and the processing module is used for processing the message according to the position information and the target mark.
The present application provides a message processing apparatus, comprising:
a processor and a machine-readable storage medium having stored thereon a plurality of computer instructions, the processor when executing the computer instructions performs:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
The present application provides a message processing apparatus, comprising:
a processor and a machine-readable storage medium having stored thereon a plurality of computer instructions, the processor when executing the computer instructions performs:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
and processing the message according to the position information and the target mark.
Based on the technical scheme, in the embodiment of the application, the message can be stored in the data queue, the position information of the message in the data queue is determined, the message is sent to the receiver, the target mark corresponding to the message is determined, and the corresponding relation between the target mark and the position information is stored in the operation queue; then, when processing the message in the data queue, based on the position information of the message in the data queue, obtaining the target mark corresponding to the position information from the operation queue, and processing the message according to the target mark.
In the above manner, when the middleware performs persistence processing on the message, the message is stored in the data queue, instead of the message being stored in the relational database, so that the problems of high cost, generation of a large amount of random IO, influence on stability and the like of the relational database are avoided. Moreover, the above-mentioned mode can realize reliable message push, guarantee the receiver can receive the message, add to the queue sequentially to the storage of the message, upgrade, delete operation, the operation to the magnetic disc is sequential read and write, the time complexity is lower, can fully utilize the bandwidth of the magnetic disc, realize the extremely high throughput, thus has realized the message storage scheme of the low-cost high performance.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present application.
FIG. 1 is a schematic diagram of an application scenario in an embodiment of the present application;
FIG. 2 is a flow diagram of a message processing method in one embodiment of the present application;
FIG. 3 is a flow chart of a message processing method in another embodiment of the present application;
fig. 4 is a block diagram of a message processing apparatus according to an embodiment of the present application;
fig. 5 is a block diagram of a message processing device according to another embodiment of the present application.
Detailed Description
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
In the embodiment of the present application, a message processing method is provided, where the method may be applied to a system including a sender, a middleware, and a receiver, for example, the method may be a distributed system that needs reliable data transmission, as shown in fig. 1, which is an application scenario diagram of this embodiment, in fig. 1, 1 sender, 1 middleware, and 3 receivers are taken as examples, in an actual application, the number of senders may also be other numbers, the number of middleware may also be other numbers, and the number of receivers may also be other numbers, which is not limited in this respect.
The sender may also be referred to as a producer, and is a client that generates a message, and may send the message to the middleware. For example, in a distributed system, the sender may be a commodity trading platform that may perform user's operations for orders, payments, etc., and generate messages related to those operations and send the messages to the middleware.
The middleware can also be called as message middleware, can perform communication among different platforms, shields the characteristics among the platforms and protocols, realizes the cooperation among application programs, provides asynchronous decoupling for each subsystem, provides synchronous and asynchronous connection between a sender and a receiver, and performs storage and forwarding of messages.
The receiver may also be referred to as a consumer or a subscriber, and is a client for consuming a message generated by the sender, and may receive the message from the middleware and process the message according to the message. For example, in a distributed system, the recipient may be a logistics platform, which may determine a logistics state from the message; the receiver can be a data analysis platform, and the data analysis platform can analyze the personalized data of the user according to the message and then provide better service for the user by utilizing the personalized data; the receiver can be a point processing platform, and the point processing platform can distribute points for the user according to the message, so that the user can obtain better service experience. Of course, the above are just a few examples of the receiver, and the type of the receiver is not limited thereto.
The message transmitted among the sender, the middleware, and the receiver may include a message header, a message body, a message attribute, a message submission status, and other contents, and the message body carries actual data, such as a product identifier, a product price, buyer information, seller information, and the like, and the contents carried by the message are not limited.
In this embodiment, the middleware may maintain a Data Queue (Data Queue) and an operation Queue (OperateQueue), where the Data Queue may include, but is not limited to, a Queue stored in a sequential appending manner, and the operation Queue may include, but is not limited to, a Queue stored in a sequential appending manner. For example, the data queue may use a queue based on a queue storage engine, and the operation queue may also use a queue based on a queue storage engine. Of course, the queue storage engine is only an example, and is not limited to this, and the data queue and the operation queue may be implemented by any queue that stores data in an order appending manner, and the queue storage engine is taken as an example in the following.
The queue storage engine is a storage system providing a queue data structure, and based on a data queue and an operation queue of the queue storage engine, data reading and writing are mainly performed in a sequential IO (input/output) mode, that is, when data is written into the data queue or the operation queue, the data is written into the tail of the data queue or the operation queue, and when the data is read from the data queue or the operation queue, the data is read from the head of the data queue or the operation queue.
The queue storage engine may include, but is not limited to: a queue storage engine based on KAFKA (distributed publish-subscribe messaging system), a queue storage engine based on rockmq (distributed messaging middleware system). Of course, the above are just two examples of the queue storage engine, and no limitation is made to this.
In this embodiment, the data queue and the operation queue may be disposed in different storage queues, that is, the data queue and the operation queue are implemented by two storage queues. Alternatively, the data queue and the operation queue may be disposed in the same storage queue, i.e., the data queue and the operation queue are implemented by one storage queue.
For example, the middleware may generate a storage queue 1 and a storage queue 2, which are two different storage queues, implement a data queue through the storage queue 1, and implement an operation queue through the storage queue 2, that is, a subsequent write operation and a subsequent read operation to the data queue are both operations to the storage queue 1, and a subsequent write operation and a subsequent read operation to the operation queue are both operations to the storage queue 2.
For another example, the middleware may generate a store queue a, and implement a data queue and an operation queue through the store queue a, that is, a write operation and a read operation for the data queue are operations for the store queue a, a write operation and a read operation for the operation queue, and are also operations for the store queue a.
When the data queue and the operation queue are implemented by one storage queue, in order to distinguish the data queue and the operation queue in the storage queue, the storage queue may further include a type flag, where the type flag may be a data type or an operation type, the data type indicates the data queue in the storage queue, and the operation type indicates the operation queue in the storage queue. That is, the storage queue a may be divided into two parts, one part is the storage queue a1, the storage queue a1 represents a data queue in the storage queue a, and the data queue in the storage queue a may be represented by a data type; the other part is storage queue a2, storage queue a2 represents the queue of operations in storage queue a, and the queue of operations in storage queue a can be represented by the type of operations.
For example, each time data is written to the store queue a, it is possible to distinguish whether the data is written to the data queue or the operation queue; if the data is written into the data queue, adding a data type to the data, which means that the data is written into the data queue, namely into a storage queue A1; if the data is to be written to the operation queue, then an operation type is added for the data, indicating that the data is to be written to the operation queue, i.e., to store queue A2.
In addition, when data is read from the storage queue A each time, the data can be read from the data queue or the operation queue; if data is read from the data queue, data with a data type can be read, thereby indicating that data is read from the data queue, that is, data can be read from the storage queue a 1; if data is read from the operation queue, data having the operation type may be read, thereby indicating that data is read from the operation queue, i.e., data may be read from store queue a 2.
Of course, the data queue and the operation queue are implemented by one storage queue, or the data queue and the operation queue are implemented by two storage queues, which are only two examples of the present application, and this is not limited to this, and for convenience of description, the data queue and the operation queue implemented by two storage queues are taken as an example for explanation.
In the application scenario, referring to fig. 2, which is a flowchart of a message processing method in the embodiment of the present application, the message processing method may be applied to middleware, and the message processing method may include:
Specifically, after the sender sends the message to the middleware, the middleware may perform persistence processing on the message, and when performing persistence processing on the message, the middleware stores the message in the data queue instead of storing the message in the relational database. The middleware may also reply with an ACK (Acknowledgement character) to the sender indicating that the middleware has successfully received the message.
In one example, after the middleware stores the message in the data queue, the middleware can determine the position information of the message in the data queue, for example, the position information can be a position point. For example, assume that message 1-message 5 already exist in the data queue, the location information for message 1 is location 96, the location information for message 2 is location 97, the location information for message 3 is location 98, the location information for message 4 is location 99, and the location information for message 5 is location 100. Further, assuming that the current message is message 6, after the middleware stores message 6 in the data queue, it can assign site 101 to message 6, and the position information of message 6 in the data queue is site 101.
The position range of the data queue can be 1-positive infinity, and each time a new message is stored in the data queue, 1 can be added on the basis of the position of the last message in the data queue to obtain a new position, and the new position is used as the position information of the new message. For example, add 1 to the position 100 of the last message 5 in the data queue to obtain the position 101, and use the position 101 as the position information of the message 6 in the data queue.
The position range of the data queue may also be 1-a specific value, such as a specific value of 1000, so that the position information of the message is at most position 1000, and after the position information is position 1000, when a new message is stored in the data queue, the position information of the new message is position 1, that is, the bit is reallocated from the beginning.
In this case, the specific value may be equal to or greater than the maximum number of messages in the data queue to ensure that each message in the data queue has a different location, i.e., the location information of each message may be different.
Wherein, the position information of the first message (i.e. message 1) in the data queue is position 96, which indicates that the position information is the messages from position 1 to position 95, and has been processed and deleted from the data queue.
In the above embodiment, when the middleware stores the message in the data queue, the message is stored in the tail of the data queue. For example, assuming that messages 1-5 already exist in the data queue, message 5 is the last message of the data queue, then when the current message is message 6, message 6 is stored to the back of message 5.
The message in this embodiment is used to update any field of the message, which is not limited to this.
After the sender sends the message to the middleware, the middleware can also send the message to the receiver. When the middleware sends the message to the receiver, the message is not read from the data queue and sent to the receiver, but the following method is adopted: after receiving the message, the middleware stores the message into a memory, then reads the message from the memory, stores the message into a data queue, reads the message from the memory and sends the message to a receiver; the message may then be deleted from memory.
Wherein the middleware sends the message to the recipient, which may include but is not limited to: the middleware analyzes the theme parameters from the message, queries the mapping table through the theme parameters to obtain the receiver identification corresponding to the theme parameters, and then can send the message to the receiver corresponding to the receiver identification.
For example, the middleware may record the corresponding relationship between the topic parameter a and the recipient identifier 1, the recipient identifier 2, and the recipient identifier 3 in the mapping table, and the establishment of the mapping table is not limited as long as the mapping table can record the corresponding relationship between the topic parameter and the recipient identifier. Further, assuming that the subject parameter carried in the message is the subject parameter a, the middleware may query the mapping table through the subject parameter a to obtain the recipient identifier 1, the recipient identifier 2, and the recipient identifier 3 corresponding to the subject parameter a. The middleware may then send the message to recipient 1 corresponding to recipient identification 1, to recipient 2 corresponding to recipient identification 2, and to recipient 3 corresponding to recipient identification 3.
Further, if the middleware receives the ACK returned by the receiver 1 aiming at the message within the preset time, the message of the receiver 1 is determined to be successfully sent; if the middleware receives the ACK returned by the receiver 2 aiming at the message within the preset time, the message of the receiver 2 is determined to be successfully sent; if the middleware does not receive the ACK returned by the receiver 3 for the message within the preset time, it is determined that the message transmission of the receiver 3 fails.
In one example, when the middleware sends the message to the receiver, the message can be sent to the receiver in a push (push) mode or a pull (pull) mode. For example, in the push mode, the middleware may send the message to the receiver by adopting the mode of the above embodiment each time the middleware receives the message sent by the sender. In the pull mode, after receiving the request of the receiver, the middleware may send the message to the receiver by using the above-described embodiment. Of course, the above is only an example of message sending, and is not limited thereto.
In one example, determining the target tag corresponding to the message may include, but is not limited to: and determining the target mark corresponding to the message according to the sending condition of the message. Specifically, if the sending condition is that the message is successfully sent (i.e. all the recipients' messages have been successfully sent), the target mark may be determined to be a deletion mark. Alternatively, if the sending condition is a message sending failure (i.e. a message sending failure of any one or more recipients), the target mark may be an update mark and a recipient mark, where the recipient mark may include an identifier of the recipient who failed to send the message. Of course, the above is only an example of determining the target mark, and this is not limited thereto, for example, in some application scenarios, if the message needs to be retransmitted regardless of success or failure of message transmission, it may be directly determined that the target mark is the update mark and the recipient identifier.
For example, after the middleware sends the message to the receiver 1, the receiver 2, and the receiver 3, if the message sending of the receiver 1 is successful, the message sending of the receiver 2 is successful, and the message sending of the receiver 3 is successful, it may be determined that the sending condition is the message sending success, and it may be determined that the target flag is a deletion flag, such as the flag RR.
If the message transmission of receiver 1 is successful, the message transmission of receiver 2 is successful, and the message transmission of receiver 3 is failed, it may be determined that the transmission condition is the message transmission failure, and it is determined that the target tag is an update tag (e.g., tag SS) and the identity of receiver 3, but does not include the identities of receiver 1 and receiver 2.
In one example, the middleware sends a message to the recipient, which may include but is not limited to: after the sender sends the message A to the middleware, the middleware stores the message A in a data queue and sends the message A to the receiver. Or after the sender sends the message A to the middleware, the middleware stores the message A in the data queue, and if the state of the message A is a prepare state, the middleware does not currently send the message A to the receiver. If the sender initiates a commit operation for the message a within a certain time range, the middleware may write information of the commit operation to the operation queue and update the status of the message a to a commit status, and then the middleware transmits the message a to the receiver. In addition, if the sender initiates a rollback operation on the message a within a certain time range, the middleware may write information of the rollback operation into the operation queue and update the status of the message a to the rollback status, so that the message a is equivalently deleted and the middleware no longer sends the message a to the receiver. Of course, the above are just a few examples of the middleware sending a message to a recipient, and are not limited thereto.
In this embodiment, a data queue and an operation queue may be used to separate data and operations, that is, contents related to data may be stored in the data queue, such as storing messages in the data queue, and contents related to operations may be stored in the operation queue, such as storing object tags and location information in the operation queue.
For example, in step 201, the middleware stores message 6 in a data queue, and may determine that the position information of message 6 in the data queue is site 101; in step 202, if the middleware determines that the target marker is a deletion marker (e.g., marker RR), in step 203, the middleware may store the corresponding relationship between the site 101 and the deletion marker in the operation queue; alternatively, if the middleware determines that the target tag is an update tag (e.g., tag SS) and the identity of recipient 3 in step 202, the middleware may store the correspondence of location 101, the update tag, and the identity of recipient 3 in the operation queue in step 203.
In step 204, when the message in the data queue is processed, based on the position information (such as the position point) of the message in the data queue, a target mark corresponding to the position information is obtained from the operation queue.
Specifically, each message in the data queue may be processed in sequence by a sequential processing manner starting from the first message in the data queue; when each message in the data queue is processed, the target mark corresponding to the position information is obtained from the operation queue based on the position information of the message in the data queue.
For example, after the middleware stores message 6 in the data queue, there are messages 1-6 in the data queue, and the order of these messages may be, from front to back: message 1, message 2, message 3, message 4, message 5, and message 6, then, starting with the first message (i.e., message 1) of the data queue, each message in the data queue is processed in sequence in a sequential processing manner, such as processing message 1 first, processing message 2 after the processing of message 1 is completed, processing message 3 after the processing of message 2 is completed, processing message 4 after the processing of message 3 is completed, processing message 5 after the processing of message 4 is completed, processing message 6 after the processing of message 5 is completed, and so on. Since each message is processed in the same manner, the following description will take the example of processing the message 6.
When processing a message 6 in the data queue, based on the position information (e.g., position 101) of the message 6 in the data queue, a target tag corresponding to the position information can be obtained from the operation queue. Specifically, since the operation queue is used to store the correspondence between the location information and the target mark, based on the location information (e.g., locus 101), the target mark corresponding to the location information, such as the target mark is a deletion mark (e.g., mark RR), or the mark (e.g., mark SS) and the identifier of the receiver 3 are updated.
In the above embodiment, step 201-step 203 may be implemented by the same process (e.g., process a), and step 204-step 205 may be implemented by another process (e.g., process B), where process a and process B are two different processes and the processing procedure is independent, i.e., the processing of process B is independent of process a.
In one example, processing the message according to the target mark may include, but is not limited to:
in case one, if the target is marked as a delete marker, the message is deleted from the data queue.
For example, when processing a message 6 in a data queue, if the target is marked as a deletion marker (e.g., a marker RR), the message 6 can be deleted from the data queue, and thus, the processing of the message 6 can be completed.
And in the second case, if the target mark is the updating mark and the receiver identification, deleting the message from the data queue, generating an updating message according to the message, storing the updating message into the data queue, and sending the updating message to the receiver corresponding to the receiver identification. Then, the update position information of the update message in the data queue can be determined, the update target mark is determined according to the sending condition of the update message, and the corresponding relation between the update position information and the update target mark is stored in the operation queue.
In one example, storing the update message to a data queue may further include, but is not limited to: instead of replacing the location of the message itself, the update message is additionally stored to the tail of the data queue.
For example, when processing a message 6 in a data queue, if the target is marked as an update flag (e.g. flag SS) and the identifier of the receiver 3, the message 6 may be deleted from the data queue, and an update message is generated according to the message 6, and then the update message is referred to as a message 7, where the message 7 may be the same as the message 6 or different from the message 6, without limitation, the message 7 is generated based on the message 6, and the message 7 may be a new message, and the foregoing steps 201 to 205 are re-executed for the message 7, except that: instead of sending a message 7 to all recipients, the message 7 is sent to the corresponding recipient identification in the target token.
For example, instead of replacing the location of message 6 itself, message 7 may be appended to the end of the data queue, the location information for message 7 in the data queue (i.e., the updated location information described above, such as location 125) may be determined, and message 7 may be sent to recipient 3 based on the identification of recipient 3 in the target tag, rather than sending message 7 to recipient 1 and recipient 2. Then, the target flag (i.e. the update target flag) is determined according to the sending condition of the message 7, and the specific determination manner is referred to step 202, which is not described herein again. Then, storing the corresponding relation between the target mark and the position information into an operation queue; then, step 204 and step 205 are performed for message 7, and will not be repeated here. In step 205, if the target is marked as the update flag and the recipient id, the message 7 is deleted from the data queue, and an update message is generated according to the message 7, such as message 8, and the above steps 201 to 205 are executed again for the message 8, and so on.
In an example, the execution sequence is only an example given for convenience of description, and in practical applications, the execution sequence between steps may also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the respective methods do not have to be performed in the order shown and described herein, and the methods may include more or less steps than those described herein. Moreover, a single step described in this specification may be broken down into multiple steps for description in other embodiments; multiple steps described in this specification may be combined into a single step in other embodiments.
Based on the technical scheme, in the embodiment of the application, when the middleware performs persistence processing on the message, the message is stored in the data queue instead of the relational database, so that the problems that the relational database is high in cost, generates a large amount of random IO (input/output), affects stability and the like are solved. The method can realize reliable message pushing, ensures that a receiver can receive the message, sequentially adds the operations of storing, updating and deleting the message into the queue, sequentially reads and writes the operation of the disk, has low time complexity, can fully utilize the bandwidth of the disk, realizes extremely high throughput, and further realizes a message storage scheme with low cost and high performance. In the above manner, the middleware can maintain the message state, record the receivers with failed sending, and periodically perform message re-pushing, thereby ensuring that each receiver can receive the message. By separating data and operation during writing and combining the data and operation during reading, the information is updated, and the throughput is improved.
Based on the same concept as the above method, an embodiment of the present application proposes another message processing method, which is shown in fig. 3 and is a flowchart of the method, where the method may be applied to middleware, and the method may include:
Wherein, processing the message according to the location information and the target mark may include but is not limited to: storing the corresponding relation between the target mark and the position information into an operation queue; then, when the message in the data queue is processed, based on the position information of the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue; the message is then processed according to the target mark.
The flow of step 301 to step 303 may refer to step 201 to step 205, which is not described herein again.
In an example, the execution sequence is only an example given for convenience of description, and in practical applications, the execution sequence between steps may also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the respective methods do not have to be performed in the order shown and described herein, and the methods may include more or less steps than those described herein. Moreover, a single step described in this specification may be broken down into multiple steps for description in other embodiments; multiple steps described in this specification may be combined into a single step in other embodiments.
Based on the technical scheme, in the embodiment of the application, when the middleware performs persistence processing on the message, the message is stored in the data queue instead of the relational database, so that the problems that the relational database is high in cost, generates a large amount of random IO (input/output), affects stability and the like are solved. The method can realize reliable message pushing, ensures that a receiver can receive the message, sequentially adds the operations of storing, updating and deleting the message into the queue, sequentially reads and writes the operation of the disk, has low time complexity, can fully utilize the bandwidth of the disk, realizes extremely high throughput, and further realizes a message storage scheme with low cost and high performance. In the above manner, the middleware can maintain the message state, record the receivers with failed sending, and periodically perform message re-pushing, thereby ensuring that each receiver can receive the message. By separating data and operation during writing and combining the data and operation during reading, the information is updated, and the throughput is improved.
Based on the same application concept as the method described above, an embodiment of the present application further provides a message processing apparatus, as shown in fig. 4, where the apparatus is a structural diagram of the message processing apparatus, and the apparatus may include:
a determining module 41, configured to store a message in a data queue, and determine location information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message;
a storage module 42, configured to store the relationship between the target mark and the location information in an operation queue;
an obtaining module 43, configured to, when processing a message in the data queue, obtain, based on location information of the message in the data queue, a target mark corresponding to the location information from the operation queue;
and the processing module 44 is configured to process the message according to the target tag.
The determining module 41 is specifically configured to, when determining the target mark according to the sending condition of the message:
if the sending condition is that the message is successfully sent, determining that the target mark is a deletion mark; alternatively, the first and second electrodes may be,
and if the sending condition is that the message sending fails, determining that the target mark is an updating mark and a receiver mark, wherein the receiver mark comprises the mark of a receiver of which the message sending fails.
When the processing module 44 determines the target mark corresponding to the message, it is specifically configured to:
determining a target mark corresponding to the message according to the sending condition of the message; wherein:
if the target mark is a deletion mark, deleting the message from the data queue; alternatively, the first and second electrodes may be,
if the target mark is an update mark and a receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver mark; determining update location information of the update message in the data queue; determining an update target mark according to the sending condition of the update message; and storing the corresponding relation between the updated position information and the updated target mark in an operation queue.
Based on the same application concept as the method, an embodiment of the present application further provides a message processing apparatus, including: a processor and a machine-readable storage medium; wherein the machine-readable storage medium has stored thereon a plurality of computer instructions, and the processor executes the computer instructions to perform the following:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
An embodiment of the present application further provides a machine-readable storage medium, where a number of computer instructions are stored on the machine-readable storage medium, and when executed, the computer instructions perform the following processes:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
Based on the same application concept as the method described above, an embodiment of the present application further provides a message processing apparatus, as shown in fig. 5, where the apparatus is a structural diagram of the message processing apparatus, and the apparatus may include:
a determining module 51, configured to store a message in a data queue, and determine location information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message;
and a processing module 52, configured to process the message according to the location information and the target mark.
Based on the same application concept as the method, an embodiment of the present application further provides a message processing apparatus, including: a processor and a machine-readable storage medium; wherein the machine-readable storage medium has stored thereon a plurality of computer instructions, and the processor executes the computer instructions to perform the following:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
and processing the message according to the position information and the target mark.
An embodiment of the present application further provides a machine-readable storage medium, where a number of computer instructions are stored on the machine-readable storage medium, and when executed, the computer instructions perform the following processes:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
and processing the message according to the position information and the target mark.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (18)
1. A method of message processing, the method comprising:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
2. The method of claim 1,
the data queue comprises a queue which stores in a sequential addition mode;
the operation queue comprises a queue which stores in a sequential addition mode.
3. The method of claim 1,
the data queue and the operation queue are deployed in different storage queues; alternatively, the first and second electrodes may be,
the data queue and the operation queue are deployed in the same storage queue, the storage queue comprises a type mark, and the type mark is a data type or an operation type; the data type represents a data queue in the storage queue, and the operation type represents an operation queue in the storage queue.
4. The method of claim 1,
the determining the target mark corresponding to the message includes:
and determining a target mark corresponding to the message according to the sending condition of the message.
5. The method of claim 4,
the determining the target mark corresponding to the message according to the sending condition of the message includes:
if the sending condition is that the message is successfully sent, determining that the target mark is a deletion mark;
and if the sending condition is that the message sending fails, determining that the target mark is an updating mark and a receiver mark, wherein the receiver mark comprises the mark of a receiver of which the message sending fails.
6. The method of claim 1,
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue, including:
sequentially processing each message in the data queue by adopting a sequential processing mode from the first message of the data queue; when the message in the data queue is processed, the target mark corresponding to the position information is obtained from the operation queue based on the position information of the message in the data queue.
7. The method according to claim 1 or 5,
the processing the message according to the target mark comprises:
and if the target mark is a deletion mark, deleting the message from the data queue.
8. The method according to claim 1 or 5,
the processing the message according to the target mark comprises:
and if the target mark is an update mark and a receiver identifier, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver identifier.
9. The method of claim 8,
the storing the update message to the data queue includes:
and additionally storing the updating message to the tail part of the data queue.
10. The method of claim 8,
after storing the update message to the data queue, the method further comprises:
determining update location information of the update message in the data queue;
determining an update target mark according to the sending condition of the update message;
and storing the corresponding relation between the updated position information and the updated target mark in an operation queue.
11. A method of message processing, the method comprising:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
and processing the message according to the position information and the target mark.
12. The method of claim 11,
the processing the message according to the location information and the target mark includes:
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
13. A message processing apparatus, characterized in that the apparatus comprises:
the determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message;
the storage module is used for storing the relation between the target mark and the position information to an operation queue;
the acquisition module is used for acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue when the message in the data queue is processed;
and the processing module is used for processing the message according to the target mark.
14. The apparatus of claim 13,
the determining module, when determining the target mark corresponding to the message, is specifically configured to:
determining a target mark corresponding to the message according to the sending condition of the message; wherein:
if the sending condition is that the message is successfully sent, determining that the target mark is a deletion mark; alternatively, the first and second electrodes may be,
and if the sending condition is that the message sending fails, determining that the target mark is an updating mark and a receiver mark, wherein the receiver mark comprises the mark of a receiver of which the message sending fails.
15. The apparatus of claim 13 or 14,
the processing module is specifically configured to, when processing the message according to the target tag:
if the target mark is a deletion mark, deleting the message from the data queue; alternatively, the first and second electrodes may be,
if the target mark is an update mark and a receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver mark; determining update location information of the update message in the data queue; determining an update target mark according to the sending condition of the update message; and storing the corresponding relation between the updated position information and the updated target mark in an operation queue.
16. A message processing apparatus, characterized in that the apparatus comprises:
the determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; sending the message to a receiver, and determining a target mark corresponding to the message;
and the processing module is used for processing the message according to the position information and the target mark.
17. A message processing device, comprising:
a processor and a machine-readable storage medium having stored thereon a plurality of computer instructions, the processor when executing the computer instructions performs:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
storing the corresponding relation between the target mark and the position information into an operation queue;
when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
18. A message processing device, comprising:
a processor and a machine-readable storage medium having stored thereon a plurality of computer instructions, the processor when executing the computer instructions performs:
storing a message to a data queue, and determining the position information of the message in the data queue;
sending the message to a receiver, and determining a target mark corresponding to the message;
and processing the message according to the position information and the target mark.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811142226.XA CN110968431B (en) | 2018-09-28 | 2018-09-28 | Message processing method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811142226.XA CN110968431B (en) | 2018-09-28 | 2018-09-28 | Message processing method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110968431A true CN110968431A (en) | 2020-04-07 |
| CN110968431B CN110968431B (en) | 2023-06-20 |
Family
ID=70027035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811142226.XA Active CN110968431B (en) | 2018-09-28 | 2018-09-28 | Message processing method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110968431B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111970091A (en) * | 2020-08-06 | 2020-11-20 | 北京浪潮数据技术有限公司 | Communication method, system, equipment and computer readable storage medium |
| CN111988391A (en) * | 2020-08-14 | 2020-11-24 | 网联清算有限公司 | Message sending method and device |
| CN112069438A (en) * | 2020-09-10 | 2020-12-11 | 北京五八信息技术有限公司 | Message merging method, message processing method and device, equipment and medium |
| CN112134938A (en) * | 2020-09-09 | 2020-12-25 | 重庆易宠科技有限公司 | Message processing method, device, terminal and computer readable storage medium |
| CN112286973A (en) * | 2020-11-19 | 2021-01-29 | 每日互动股份有限公司 | Data message storage method and device, computer equipment and storage medium |
| CN112925642A (en) * | 2021-02-25 | 2021-06-08 | 百果园技术(新加坡)有限公司 | Delayed message processing method, device, equipment and storage medium |
| CN113296976A (en) * | 2021-02-10 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Message processing method, message processing device, electronic equipment, storage medium and program product |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120005286A1 (en) * | 2010-04-27 | 2012-01-05 | International Business Machines Corporation | Application of System Level Policy in Message Oriented Middleware |
| CN102469033A (en) * | 2010-11-01 | 2012-05-23 | 阿里巴巴集团控股有限公司 | Message subscription system and message sending method |
| CN104731912A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Message transmission method and device for message middleware MQ |
| CN105242975A (en) * | 2015-08-27 | 2016-01-13 | 浪潮软件股份有限公司 | Message transmission method and message middleware |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN107133302A (en) * | 2017-04-28 | 2017-09-05 | 努比亚技术有限公司 | Realize method, system, terminal and the computer-readable recording medium of data consistency |
| CN108009027A (en) * | 2017-11-23 | 2018-05-08 | 北京百度网讯科技有限公司 | Implementation method, device, equipment and the storage medium of queue message uniformity |
| CN108170545A (en) * | 2018-01-08 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of method for message transmission and device based on message-oriented middleware |
-
2018
- 2018-09-28 CN CN201811142226.XA patent/CN110968431B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120005286A1 (en) * | 2010-04-27 | 2012-01-05 | International Business Machines Corporation | Application of System Level Policy in Message Oriented Middleware |
| CN102469033A (en) * | 2010-11-01 | 2012-05-23 | 阿里巴巴集团控股有限公司 | Message subscription system and message sending method |
| CN104731912A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Message transmission method and device for message middleware MQ |
| CN105242975A (en) * | 2015-08-27 | 2016-01-13 | 浪潮软件股份有限公司 | Message transmission method and message middleware |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN107133302A (en) * | 2017-04-28 | 2017-09-05 | 努比亚技术有限公司 | Realize method, system, terminal and the computer-readable recording medium of data consistency |
| CN108009027A (en) * | 2017-11-23 | 2018-05-08 | 北京百度网讯科技有限公司 | Implementation method, device, equipment and the storage medium of queue message uniformity |
| CN108170545A (en) * | 2018-01-08 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of method for message transmission and device based on message-oriented middleware |
Non-Patent Citations (4)
| Title |
|---|
| HE MING YAO DEPARTMENT OF ELECTRICAL AND ELECTRONIC ENGINEERING, THE UNIVERSITY OF HONG KONG, HONG KONG等: "Machine learning based method of moments (ML-MoM)" * |
| 史冬冬: "云队列：一个基于Hadoop的大规模消息基础平台" * |
| 汪添生;尉双梅;崔蔚;刘迪;何金陵;孙琦;: "支持MongoDB的事务管理方案研究" * |
| 许静;牟艳;秦江龙;: "竞赛数据交换平台的研究与实现" * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111970091A (en) * | 2020-08-06 | 2020-11-20 | 北京浪潮数据技术有限公司 | Communication method, system, equipment and computer readable storage medium |
| CN111988391A (en) * | 2020-08-14 | 2020-11-24 | 网联清算有限公司 | Message sending method and device |
| WO2022033586A1 (en) * | 2020-08-14 | 2022-02-17 | 网联清算有限公司 | Message sending method and device |
| CN112134938A (en) * | 2020-09-09 | 2020-12-25 | 重庆易宠科技有限公司 | Message processing method, device, terminal and computer readable storage medium |
| CN112134938B (en) * | 2020-09-09 | 2022-08-26 | 重庆易宠科技有限公司 | Message processing method, device, terminal and computer readable storage medium |
| CN112069438A (en) * | 2020-09-10 | 2020-12-11 | 北京五八信息技术有限公司 | Message merging method, message processing method and device, equipment and medium |
| CN112069438B (en) * | 2020-09-10 | 2023-12-22 | 北京五八信息技术有限公司 | Message merging method, message processing method and device, equipment and medium |
| CN112286973A (en) * | 2020-11-19 | 2021-01-29 | 每日互动股份有限公司 | Data message storage method and device, computer equipment and storage medium |
| CN113296976A (en) * | 2021-02-10 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Message processing method, message processing device, electronic equipment, storage medium and program product |
| CN112925642A (en) * | 2021-02-25 | 2021-06-08 | 百果园技术(新加坡)有限公司 | Delayed message processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110968431B (en) | 2023-06-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110968431B (en) | Message processing method, device and equipment | |
| CN101686209B (en) | Method and device for storing message in message retransmission system | |
| CN110968586B (en) | Distributed transaction processing method and device | |
| EP2998863A1 (en) | Converting a serial transaction schedule to a parallel transaction schedule | |
| CN109391646B (en) | Message middleware message acquisition method, device and system | |
| US9253133B2 (en) | Message thread identification and management | |
| CN107544869B (en) | Data recovery method and device | |
| CN105468302B (en) | A kind of method, apparatus and system of processing data | |
| CN106844662B (en) | Method and device for recording reading state of instant message | |
| CN104618432B (en) | A kind of processing method and processing system that event sends and receives | |
| CN106326499A (en) | Data processing method and device | |
| CN113094362B (en) | Method and device for reliably delivering and processing asynchronous message | |
| CN111198923A (en) | Data synchronization method and system based on message queue | |
| CN102360389A (en) | Method for implementing data set specific management policies with respect to data | |
| US20200403850A1 (en) | Facilitating resilient and fault tolerant asynchronous messaging | |
| CN110071968A (en) | A kind of message storage method and device based on block chain | |
| CN110069506A (en) | Maintaining method, device and the server of business datum | |
| CN110555020B (en) | Mapping relation establishing and data querying methods, devices and equipment | |
| CN117692418A (en) | Message processing method, device, computer equipment and storage medium | |
| CN111078425B (en) | Message processing method and device, storage medium and electronic equipment | |
| CN110830361B (en) | Mail data storage method and device | |
| US20150365365A1 (en) | Method and apparatus for modifying message | |
| CN108694219B (en) | Data processing method and device | |
| CN110266610B (en) | Traffic identification method and device and electronic equipment | |
| CN113672488A (en) | Log text processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |