CN110968431B - Message processing method, device and equipment - Google Patents
Message processing method, device and equipment Download PDFInfo
- Publication number
- CN110968431B CN110968431B CN201811142226.XA CN201811142226A CN110968431B CN 110968431 B CN110968431 B CN 110968431B CN 201811142226 A CN201811142226 A CN 201811142226A CN 110968431 B CN110968431 B CN 110968431B
- Authority
- CN
- China
- Prior art keywords
- message
- queue
- mark
- data queue
- target mark
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The application provides a message processing method, a device and equipment, wherein the method comprises the following steps: storing a message into a data queue, and determining the position information of the message in the data queue; the message is sent to a receiver, and a target mark corresponding to the message is determined; storing the corresponding relation between the target mark and the position information into an operation queue; when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue; and processing the message according to the target mark. According to the technical scheme, the operations of storing, updating and deleting the information are sequentially added to the queue, the operations of the disk are sequentially read and written, the time complexity is low, the disk bandwidth can be fully utilized, and extremely high throughput is achieved.
Description
Technical Field
The present disclosure relates to the field of internet technologies, and in particular, to a method, an apparatus, and a device for processing a message.
Background
The middleware can be suitable for a distributed environment requiring reliable data transmission, and different objects activate events of the other party by transmitting messages in a system adopting the middleware, so that corresponding operations are completed. Specifically, the sender sends a message to the middleware, which forwards the message to the recipient. The middleware can communicate between different platforms, is often used for shielding characteristics among various platforms and protocols, realizes cooperation among application programs, and has the advantages of providing synchronous and asynchronous connection between a sender and a receiver, and storing and forwarding messages at any time.
After the sender sends the message to the middleware, the middleware needs to persist the message, such as storing the message in a relational database, and then forwarding the message to the recipient. Thus, once a message is not successfully forwarded to the recipient, the middleware reads the message from the relational database and retransmits the message, and so on, until the message is successfully forwarded to the recipient, the middleware deletes the message from the relational database.
However, when the middleware performs persistence processing on the message, the message is stored in a relational database, and the relational database has high cost and can generate a large number of random IOs (input and output), so that the stability is affected.
Disclosure of Invention
The application provides a message processing method, which comprises the following steps:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
storing the corresponding relation between the target mark and the position information into an operation queue;
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
And processing the message according to the target mark.
The application provides a message processing method, which comprises the following steps:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
and processing the message according to the position information and the target mark.
The application provides a message processing apparatus, the apparatus comprising:
the determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; transmitting a message to a receiver, and determining a target mark corresponding to the message;
a storage module for storing the relation between the target mark and the position information to an operation queue;
the acquisition module is used for acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue when the message in the data queue is processed;
and the processing module is used for processing the message according to the target mark.
The application provides a message processing apparatus, the apparatus comprising:
The determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; transmitting a message to a receiver, and determining a target mark corresponding to the message;
and the processing module is used for processing the message according to the position information and the target mark.
The application provides a message processing device, comprising:
a processor and a machine-readable storage medium having stored thereon computer instructions that when executed by the processor perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
storing the corresponding relation between the target mark and the position information into an operation queue;
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
The application provides a message processing device, comprising:
A processor and a machine-readable storage medium having stored thereon computer instructions that when executed by the processor perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
and processing the message according to the position information and the target mark.
Based on the above technical solution, in the embodiments of the present application, a message may be stored in a data queue, location information of the message in the data queue is determined, the message is sent to a receiver, a target mark corresponding to the message is determined, and a correspondence between the target mark and the location information is stored in an operation queue; then, when processing the message in the data queue, based on the position information of the message in the data queue, a target mark corresponding to the position information is acquired from the operation queue, and the message is processed according to the target mark.
In the mode, when the middleware performs persistence processing on the message, the message is stored in the data queue, but not in the relational database, so that the problems of high cost, large number of random IO, influence on stability and the like of the relational database are avoided. Moreover, the method can realize reliable message pushing, ensure that a receiver can receive the message, sequentially add the operations of storing, updating and deleting the message to a queue, sequentially read and write the operation of a disk, has lower time complexity, can fully utilize the disk bandwidth and realize extremely high throughput, thereby realizing a message storage scheme with low cost and high performance.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following description will briefly describe the drawings that are required to be used in the embodiments of the present application or the description in the prior art, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings may also be obtained according to these drawings of the embodiments of the present application for a person having ordinary skill in the art.
FIG. 1 is a schematic view of an application scenario in one embodiment of the present application;
FIG. 2 is a flow chart of a message processing method in one embodiment of the present application;
FIG. 3 is a flow chart of a message processing method in another embodiment of the present application;
FIG. 4 is a block diagram of a message processing apparatus in one embodiment of the present application;
fig. 5 is a block diagram of a message processing apparatus in another embodiment of the present application.
Detailed Description
The terminology used in the embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to any or all possible combinations including one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, a first message may also be referred to as a second message, and similarly, a second message may also be referred to as a first message, without departing from the scope of the present application. Depending on the context, furthermore, the word "if" used may be interpreted as "at … …" or "at … …" or "in response to a determination".
In fig. 1, taking 1 sender, 1 middleware and 3 receivers as examples, in practical application, the number of senders may also be other numbers, the number of middleware may also be other numbers, and the number of receivers may also be other numbers, which is not limited in this regard.
Wherein the sender, which may also be referred to as a producer, is a client that generates a message, and may send the message to the middleware. For example, in a distributed system, the sender may be a commodity transaction platform, may perform user's order, payment, etc., and generate messages related to these operations, and send the messages to the middleware.
The middleware can also be called message middleware, can communicate among different platforms, shield characteristics among the platforms and protocols, realize coordination among application programs, provide asynchronous decoupling for all subsystems, provide synchronous and asynchronous connection between a sender and a receiver, and store and forward messages.
Wherein a recipient, which may also be referred to as a consumer or subscriber, is a client for consuming a message generated by a sender, which message may be received from a middleware and processed in accordance with the message. For example, in a distributed system, the recipient may be a logistics platform, which may determine the logistics status from the message; the receiver can be a data analysis platform, and the data analysis platform can analyze personalized data of the user according to the message, and then provide better service for the user by utilizing the personalized data; the receiver can be an integral processing platform, and the integral processing platform can distribute the integral to the user according to the message, so that the user can obtain better service experience. Of course, the above are just a few examples of recipients, and the type of recipients is not limited in this regard.
The message transmitted among the sender, the middleware and the receiver can include a message header, a message body, a message attribute, a message submitting state and the like, wherein actual data such as commodity identification, commodity price, buyer information, seller information and the like are carried in the message body, and the content carried by the message is not limited.
In this embodiment, the middleware may maintain a Data Queue (Data Queue) and an operation Queue (operation Queue), where the Data Queue may include, but is not limited to, a Queue that stores in a sequential addition manner, and the operation Queue may include, but is not limited to, a Queue that stores in a sequential addition manner. For example, the data queue may use a queue based on a queue store engine, and the operation queue may use a queue based on a queue store engine. Of course, the queue store engine is only one example, and the data queue and the operation queue may be implemented using any queue that stores in sequential append, and the queue store engine is used as an example.
The queue storage engine is a storage system for providing a queue data structure, based on a data queue and an operation queue of the queue storage engine, data reading and writing are mainly performed in sequence IO, that is, when data is written into the data queue or the operation queue, the data is written into the tail part of the data queue or the operation queue, and when the data is read from the data queue or the operation queue, the data is read from the head part of the data queue or the operation queue.
Wherein the queue storage engine may include, but is not limited to: a KAFKA (distributed publish-subscribe messaging system) based queue storage engine, a rock mq (distributed message middleware system) based queue storage engine. Of course, the above are just two examples of queue storage engines, and are not limiting.
In this embodiment, the data queue and the operation queue may be disposed in different storage queues, that is, the data queue and the operation queue are implemented by two storage queues. Alternatively, the data queue and the operation queue may be disposed in the same store queue, i.e., the data queue and the operation queue are implemented by one store queue.
For example, the middleware may generate a store queue 1 and a store queue 2, which are two different store queues, and implement a data queue through the store queue 1, and implement an operation queue through the store queue 2, that is, the subsequent write operation and read operation for the data queue are both operations for the store queue 1, and the subsequent write operation and read operation for the operation queue are both operations for the store queue 2.
For another example, the middleware may generate the storage queue a, and implement the data queue and the operation queue through the storage queue a, that is, the write operation and the read operation for the data queue are the operations for the storage queue a, the write operation and the read operation for the operation queue are the operations for the storage queue a.
When the data queue and the operation queue are implemented by one storage queue, in order to distinguish the data queue and the operation queue in the storage queue, the storage queue may further include a type flag, which may be a data type or an operation type, the data type represents the data queue in the storage queue, and the operation type represents the operation queue in the storage queue. That is, the above-mentioned storage queue a may be divided into two parts, one part being a storage queue A1, the storage queue A1 representing the data queue in the storage queue a, and the data queue in the storage queue a may be represented by a data type; the other part is a store queue A2, the store queue A2 representing the operation queue in the store queue a, which may be represented by an operation type.
For example, each time data is written into store queue A, it may be distinguished whether the data is written into the data queue or the operation queue; if the data is written into the data queue, adding a data type into the data, wherein the data is written into the data queue, namely, the data is written into the storage queue A1; if the data is written into the operation queue, the operation type is added to the data, which means that the data is written into the operation queue, namely, the data is written into the storage queue A2.
In addition, each time data is read from the storage queue a, it is also possible to distinguish whether data is read from the data queue or data is read from the operation queue; if data is read from the data queue, data having a data type may be read, thereby indicating that data is read from the data queue, i.e., data may be read from the storage queue A1; if data is read from the operation queue, data having the operation type may be read, indicating that data is read from the operation queue, i.e. data may be read from the store queue A2.
Of course, the data queue and the operation queue are implemented by one storage queue, or the data queue and the operation queue are implemented by two storage queues, which are only two examples of the present application, and for convenience of description, the data queue and the operation queue are implemented by two storage queues.
In the above application scenario, referring to fig. 2, a flowchart of a message processing method in an embodiment of the present application is shown, where the message processing method may be applied to middleware, and the message processing method may include:
Specifically, after the sender sends the message to the middleware, the middleware can perform persistence processing on the message, and when the message is subjected to persistence processing, the middleware does not store the message in the relational database, but stores the message in the data queue. In addition, the middleware may reply an ACK (Acknowledgement character) to the sender, indicating that the middleware has successfully received the message.
In one example, after the middleware stores the message in the data queue, location information for the message in the data queue may be determined, e.g., the location information may be a location point. For example, assume that messages 1-5 already exist in the data queue, that the location information for message 1 is location 96, that for message 2 is location 97, that for message 3 is location 98, that for message 4 is location 99, and that for message 5 is location 100. Further, assuming that the current message is message 6, after the middleware stores message 6 in the data queue, site 101 may be allocated for message 6, and the location information of message 6 in the data queue is site 101.
The range of the position of the data queue can be 1-positive infinity, and 1 can be added on the basis of the position of the last message of the data queue to obtain a new position every time a new message is stored in the data queue, and the new position is used as the position information of the new message. For example, add 1 on the basis of the position 100 of the last message 5 of the data queue, obtain position 101, and regard position 101 as the position information of message 6 in the data queue.
The location range of the data queue may also be 1-a specific value, for example, the specific value is 1000, so that the location information of the message is the location 1000 at maximum, and after the location information is the location 1000, when the new message is stored in the data queue, the location information of the new message is the location 1, that is, the location is reassigned from the beginning.
In this case, the specific value may be equal to or greater than the maximum number of messages in the data queue to ensure that each message in the data queue has a different location, i.e., the location information of each message may be different.
Where the location information of the first message in the data queue (i.e., message 1) is location 96, the message indicating that the location information is location 1-location 95 has been processed and deleted from the data queue.
In the above embodiment, when the middleware stores a message in the data queue, the message is stored in the tail of the data queue. For example, assuming that message 1-message 5 already exists in the data queue, message 5 is the last message in the data queue, message 6 is stored to the back of message 5 when the current message is message 6.
The message in this embodiment is used to update any field of the message, which is not limited.
Wherein, after sending the message to the middleware, the middleware can also send the message to the receiver. When the middleware sends the message to the recipient, instead of reading the message from the data queue and sending the message to the recipient, the following is adopted: after receiving the message, the middleware stores the message in a memory, reads the message from the memory, stores the message in a data queue, reads the message from the memory, and sends the message to a receiver; the message may then be deleted from memory.
Wherein the middleware sends the message to the recipient, which may include, but is not limited to: the middleware analyzes the theme parameters from the message, queries the mapping table through the theme parameters to obtain the receiver identification corresponding to the theme parameters, and then can send the message to the receiver corresponding to the receiver identification.
For example, the middleware may record the correspondence between the theme parameter a and the recipient identifier 1, the recipient identifier 2, and the recipient identifier 3 in the mapping table, and the establishment of the mapping table is not limited as long as the mapping table can record the correspondence between the theme parameter and the recipient identifier. Further, assuming that the theme parameter carried in the message is the theme parameter a, the middleware may query the mapping table through the theme parameter a to obtain the receiver identifier 1, the receiver identifier 2, and the receiver identifier 3 corresponding to the theme parameter a. The middleware may then send the message to recipient 1 corresponding to recipient identification 1, to recipient 2 corresponding to recipient identification 2, and to recipient 3 corresponding to recipient identification 3.
Further, if the middleware receives the ACK returned by the receiver 1 for the message within the preset time, determining that the message of the receiver 1 is successfully sent; if the middleware receives the ACK returned by the receiver 2 for the message within the preset time, the message of the receiver 2 is determined to be successfully sent; if the middleware does not receive the ACK returned by the receiver 3 for the message within the preset time, determining that the message transmission of the receiver 3 fails.
In one example, when the middleware sends a message to the receiver, the message may be sent to the receiver in a push mode or a pull mode. For example, in the push manner, the middleware may send a message to the receiver in the manner of the above embodiment each time the middleware receives a message sent by the sender. In pull mode, after receiving the request of the receiver, the middleware may send the message to the receiver in the manner of the foregoing embodiment. Of course, the above is merely an example of message transmission, and is not limited in this regard.
In one example, determining the target indicia to which the message corresponds may include, but is not limited to: and determining a target mark corresponding to the message according to the sending condition of the message. Specifically, if the transmission is successful (i.e., all the receiver's messages have been successfully transmitted), the determined target flag may be a delete flag. Alternatively, if the transmission is a message transmission failure (i.e., a message transmission failure of any one or more recipients), the determination target flag may be an update flag and a recipient identification, where the recipient identification may include an identification of the recipient for which the message transmission failed. Of course, the above is merely an example of determining the target mark, and is not limited in this respect, for example, in some application scenarios, if the message needs to be retransmitted no matter the message is sent successfully or fails, it may be directly determined that the target mark is the update mark and the recipient mark.
For example, after the middleware transmits the message to the receiver 1, the receiver 2, and the receiver 3, if the message transmission of the receiver 1 is successful, the message transmission of the receiver 2 is successful, and the message transmission of the receiver 3 is successful, it may be determined that the transmission condition is the message transmission success, and it may be determined that the target flag is a delete flag, such as the flag RR.
If the message transmission of the receiver 1 is successful, the message transmission of the receiver 2 is successful, and the message transmission of the receiver 3 is failed, it can be determined that the transmission condition is the message transmission failure, and that the target mark is the update mark (such as the mark SS) and the identity of the receiver 3, but does not include the identities of the receiver 1 and the receiver 2.
In one example, the middleware sends a message to the recipient, which may include, but is not limited to: after the sender sends message a to the middleware, the middleware stores message a to the data queue and sends message a to the receiver. Alternatively, after the sender sends message a to the middleware, the middleware stores message a in the data queue, and assuming that the state of message a is a prepare state, the middleware does not currently send message a to the receiver. If the sender initiates a commit operation on message a within a certain time frame, the middleware may write the information of the commit operation to the operation queue and update the status of message a to the commit status, and then the middleware sends message a to the recipient. In addition, if the sender initiates a rollback operation on message a within a certain time range, the middleware may write the information of the rollback operation to the operation queue and update the state of message a to the rollback state, so that message a is equivalent to being deleted, and the middleware does not send message a to the receiver. Of course, the above are just a few examples of middleware sending a message to a recipient, and are not limiting.
In this embodiment, the data queue and the operation queue may be employed to separate data and operations, that is, the content related to the data may be stored in the data queue, such as a message is stored in the data queue, and the content related to the operation may be stored in the operation queue, such as a target mark and location information are stored in the operation queue.
For example, in step 201, the middleware stores message 6 to a data queue, and may determine that the location information of message 6 in the data queue is location 101; if the middleware determines that the target tag is a delete tag (e.g., tag RR) in step 202, then in step 203, the middleware may store the correspondence between the location 101 and the delete tag in the operation queue; alternatively, if the middleware determines in step 202 that the target mark is an update mark (e.g., mark SS) and the identity of the receiver 3, then in step 203, the middleware may store the correspondence of the location 101, the update mark, and the identity of the receiver 3 in the operation queue.
In step 204, when a message in the data queue is processed, based on the location information (such as a location, etc.) of the message in the data queue, a target tag corresponding to the location information is obtained from the operation queue.
Specifically, each message in the data queue can be sequentially processed in a sequential processing mode from the first message of the data queue; when each message in the data queue is processed, a target mark corresponding to the position information is acquired from the operation queue based on the position information of the message in the data queue.
For example, after the middleware stores message 6 in the data queue, there are messages 1-6 in the data queue, and the order of these messages may be sequentially from front to back: message 1, message 2, message 3, message 4, message 5 and message 6, then each message in the data queue is processed sequentially in a sequential manner starting from the first message in the data queue (i.e., message 1), such as message 1, message 2 after message 1 processing is complete, message 3 after message 2 processing is complete, message 4 after message 3 processing is complete, message 5 after message 4 processing is complete, message 6 after message 5 processing is complete, and so on. Since each message is handled in the same manner, the following description will take the message 6 as an example.
In processing the message 6 in the data queue, based on the location information (e.g., location 101) of the message 6 in the data queue, a target tag corresponding to the location information may be obtained from the operation queue. Specifically, since the operation queue is used to store the correspondence between the location information and the target mark, the target mark corresponding to the location information (e.g., the location 101) may be obtained from the operation queue, for example, the target mark is a deletion mark (e.g., the mark RR), or the mark (e.g., the mark SS) and the identification of the receiver 3 are updated.
In step 205, the message is processed according to the target mark. For example, when processing message 6, after the target tag is obtained from the operation queue, message 6 may be processed using the target tag.
In the above embodiment, steps 201-203 may be implemented by the same process (e.g., process a), while steps 204-205 may be implemented by another process (e.g., process B), where process a and process B are two different processes, the processes are independent, i.e., the process of process B is independent of process a.
In one example, processing the message according to the target mark may include, but is not limited to:
in case one, if the target tag is a delete tag, the message is deleted from the data queue.
For example, when processing a message 6 in a data queue, if the target tag is a delete tag (e.g., tag RR), the message 6 may be deleted from the data queue, and the processing of the message 6 may be completed.
And in the second case, if the target mark is the update mark and the receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message in the data queue, and transmitting the update message to the receiver corresponding to the receiver mark. Then, the update position information of the update message in the data queue can be determined, the update target mark is determined according to the sending condition of the update message, and the corresponding relation between the update position information and the update target mark is stored in the operation queue.
In one example, storing the update message to the data queue may also include, but is not limited to: the update message is additionally stored to the tail of the data queue instead of replacing the location of the message itself.
For example, in processing message 6 in the data queue, if the target tag is an update tag (e.g., tag SS) and the identity of recipient 3, then message 6 may be deleted from the data queue and an update message may be generated based on message 6, which is subsequently referred to as message 7, which may be the same as message 6 or different from message 6, without limitation, message 7 being generated based on message 6, and message 7 may be a new message, steps 201-205 described above being re-performed for message 7, except that: instead of sending message 7 to all recipients, message 7 is sent to the recipient whose recipient identity in the target token corresponds.
For example, instead of replacing the location of message 6 itself, message 7 may be additionally stored at the end of the data queue, the location information of message 7 in the data queue (i.e., the updated location information described above, such as location 125) may be determined, and message 7 may be sent to recipient 3 based on the identity of recipient 3 in the destination tag, without sending message 7 to recipient 1 and recipient 2. Then, the target mark (i.e. the updated target mark) is determined according to the sending condition of the message 7, and the specific determining manner is referred to step 202, which is not described herein. Then, storing the corresponding relation between the target mark and the position information into an operation queue; then, step 204 and step 205 are performed for message 7, and the detailed description is not repeated here. When step 205 is performed, if the target tag is the update tag and the recipient tag, then the message 7 is deleted from the data queue, and an update message, such as message 8, is generated based on the message 7, steps 201-205 described above are re-performed for message 8, and so on.
In one example, the above execution sequence is only given for convenience of description, and in practical application, the execution sequence between steps may be changed, which is not limited. Moreover, in other embodiments, the steps of the corresponding methods need not be performed in the order shown and described herein, and the methods may include more or less steps than described herein. Furthermore, individual steps described in this specification, in other embodiments, may be described as being split into multiple steps; various steps described in this specification, in other embodiments, may be combined into a single step.
Based on the technical scheme, in the embodiment of the application, when the middleware carries out persistence processing on the message, the message is stored in the data queue, but not in the relational database, so that the problems of high cost, large number of random IO generation, stability influence and the like of the relational database are avoided. The method can realize reliable message pushing, ensure that a receiver can receive the message, sequentially add the storage, update and deletion operations of the message to the queue, sequentially read and write the operation of the disk, has lower time complexity, can fully utilize the disk bandwidth and realize extremely high throughput, thereby realizing a message storage scheme with low cost and high performance. In the mode, the middleware can maintain the message state, record the receivers with failed transmission, and regularly push the message again, so that each receiver can receive the message. By separating data from operation during writing and combining the data and the operation during reading, the information is updated, and the throughput is improved.
Based on the same concept as the above method, another message processing method is provided in the embodiment of the present application, and referring to fig. 3, a flowchart of the method is shown, and the method may be applied to middleware, and the method may include:
And step 303, processing the message according to the position information and the target mark.
Wherein the processing of the message based on the location information and the target mark may include, but is not limited to: storing the corresponding relation between the target mark and the position information into an operation queue; then, when processing the message in the data queue, based on the position information of the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue; the message is then processed according to the target mark.
The flow of steps 301 to 303 can be referred to above in steps 201 to 205, and will not be described herein.
In one example, the above execution sequence is only given for convenience of description, and in practical application, the execution sequence between steps may be changed, which is not limited. Moreover, in other embodiments, the steps of the corresponding methods need not be performed in the order shown and described herein, and the methods may include more or less steps than described herein. Furthermore, individual steps described in this specification, in other embodiments, may be described as being split into multiple steps; various steps described in this specification, in other embodiments, may be combined into a single step.
Based on the technical scheme, in the embodiment of the application, when the middleware carries out persistence processing on the message, the message is stored in the data queue, but not in the relational database, so that the problems of high cost, large number of random IO generation, stability influence and the like of the relational database are avoided. The method can realize reliable message pushing, ensure that a receiver can receive the message, sequentially add the storage, update and deletion operations of the message to the queue, sequentially read and write the operation of the disk, has lower time complexity, can fully utilize the disk bandwidth and realize extremely high throughput, thereby realizing a message storage scheme with low cost and high performance. In the mode, the middleware can maintain the message state, record the receivers with failed transmission, and regularly push the message again, so that each receiver can receive the message. By separating data from operation during writing and combining the data and the operation during reading, the information is updated, and the throughput is improved.
Based on the same application concept as the above method, the embodiment of the present application further provides a message processing apparatus, as shown in fig. 4, which is a structure diagram of the message processing apparatus, and the apparatus may include:
A determining module 41, configured to store a message in a data queue, and determine location information of the message in the data queue; transmitting a message to a receiver, and determining a target mark corresponding to the message;
a storage module 42 for storing the relationship between the target mark and the position information in an operation queue;
an obtaining module 43, configured to obtain, when processing a message in the data queue, a target tag corresponding to the location information from the operation queue based on the location information of the message in the data queue;
a processing module 44, configured to process the message according to the target mark.
The determining module 41 is specifically configured to, when determining the target mark according to the sending condition of the message:
if the sending condition is that the message sending is successful, determining that the target mark is a deletion mark; or alternatively, the process may be performed,
and if the sending condition is message sending failure, determining the target mark as an update mark and a receiver mark, wherein the receiver mark comprises the mark of a receiver with the message sending failure.
The processing module 44 is specifically configured to, when determining the target mark corresponding to the message:
determining a target mark corresponding to the message according to the sending condition of the message; wherein:
Deleting the message from the data queue if the target mark is a delete mark; or alternatively, the process may be performed,
if the target mark is an update mark and a receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver mark; determining update location information of the update message in the data queue; determining an update target mark according to the transmission condition of the update message; and storing the corresponding relation between the update position information and the update target mark into an operation queue.
Based on the same application concept as the method, the embodiment of the application further provides a message processing device, which includes: a processor and a machine-readable storage medium; wherein the machine-readable storage medium has stored thereon a number of computer instructions which, when executed by the processor, perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
Storing the corresponding relation between the target mark and the position information into an operation queue;
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
Embodiments of the present application also provide a machine-readable storage medium having stored thereon a number of computer instructions that when executed perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
storing the corresponding relation between the target mark and the position information into an operation queue;
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
and processing the message according to the target mark.
Based on the application concept same as the method, the embodiment of the application further provides a message processing device, as shown in fig. 5, which is a structure diagram of the message processing device, and the device may include:
A determining module 51, configured to store a message in a data queue, and determine location information of the message in the data queue; transmitting a message to a receiver, and determining a target mark corresponding to the message;
a processing module 52, configured to process the message according to the location information and the target mark.
Based on the same application concept as the method, the embodiment of the application further provides a message processing device, which includes: a processor and a machine-readable storage medium; wherein the machine-readable storage medium has stored thereon a number of computer instructions which, when executed by the processor, perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
and processing the message according to the position information and the target mark.
Embodiments of the present application also provide a machine-readable storage medium having stored thereon a number of computer instructions that when executed perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
The message is sent to a receiver, and a target mark corresponding to the message is determined;
and processing the message according to the position information and the target mark.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present application.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Moreover, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (13)
1. A method of message processing, the method comprising:
storing a message into a data queue, and determining the position information of the message in the data queue;
the message is sent to a receiver, and a target mark corresponding to the message is determined;
storing the corresponding relation between the target mark and the position information into an operation queue;
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
Processing the message according to the target mark; wherein said processing said message according to said target mark comprises: and if the target mark is an update mark and a receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver mark.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the data queues comprise queues which are stored in a sequential addition mode;
the operation queue comprises a queue which is stored in a sequential addition mode.
3. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the data queue and the operation queue are deployed in different storage queues; or alternatively, the process may be performed,
the data queue and the operation queue are deployed in the same storage queue, the storage queue comprises a type mark, and the type mark is a data type or an operation type; the data type represents a data queue in the store queue and the operation type represents an operation queue in the store queue.
4. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the determining the target mark corresponding to the message comprises the following steps:
and determining a target mark corresponding to the message according to the sending condition of the message.
5. The method of claim 4, wherein the step of determining the position of the first electrode is performed,
the determining the target mark corresponding to the message according to the sending condition of the message comprises the following steps:
if the sending condition is that the message sending is successful, determining that the target mark is a deletion mark;
and if the sending condition is message sending failure, determining the target mark as an update mark and a receiver mark, wherein the receiver mark comprises the mark of a receiver with the message sending failure.
6. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the step of obtaining, when processing the message in the data queue, a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue, includes:
sequentially processing each message in the data queue by adopting a sequential processing mode from the first message of the data queue; and when the message in the data queue is processed, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue.
7. The method according to claim 1 or 5, wherein,
the processing the message according to the target mark comprises the following steps:
and if the target mark is a deletion mark, deleting the message from the data queue.
8. The method of claim 1, wherein the step of determining the position of the substrate comprises,
the storing the update message to the data queue includes:
and additionally storing the update message to the tail of the data queue.
9. The method of claim 1, wherein the step of determining the position of the substrate comprises,
after the storing the update message to the data queue, the method further comprises:
determining update location information of the update message in the data queue;
determining an update target mark according to the transmission condition of the update message;
and storing the corresponding relation between the update position information and the update target mark into an operation queue.
10. A message processing apparatus, the apparatus comprising:
the determining module is used for storing the message into a data queue and determining the position information of the message in the data queue; transmitting a message to a receiver, and determining a target mark corresponding to the message;
A storage module for storing the relation between the target mark and the position information to an operation queue;
the acquisition module is used for acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue when the message in the data queue is processed;
the processing module is used for processing the message according to the target mark;
the processing module is specifically configured to, when processing the message according to the target mark: and if the target mark is an update mark and a receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver mark.
11. The apparatus of claim 10, wherein the device comprises a plurality of sensors,
the determining module is specifically configured to, when determining the target mark corresponding to the message:
determining a target mark corresponding to the message according to the sending condition of the message; wherein:
if the sending condition is that the message sending is successful, determining that the target mark is a deletion mark; or alternatively, the process may be performed,
And if the sending condition is message sending failure, determining the target mark as an update mark and a receiver mark, wherein the receiver mark comprises the mark of a receiver with the message sending failure.
12. The device according to claim 10 or 11, wherein,
the processing module is specifically configured to, when processing the message according to the target mark: deleting the message from the data queue if the target mark is a delete mark;
the processing module is further configured to, after sending the update message to the receiver corresponding to the receiver identifier: determining update location information of the update message in the data queue; determining an update target mark according to the transmission condition of the update message; and storing the corresponding relation between the update position information and the update target mark into an operation queue.
13. A message processing apparatus, comprising:
a processor and a machine-readable storage medium having stored thereon computer instructions that when executed by the processor perform the following:
storing a message into a data queue, and determining the position information of the message in the data queue;
The message is sent to a receiver, and a target mark corresponding to the message is determined;
storing the corresponding relation between the target mark and the position information into an operation queue;
when processing the message in the data queue, acquiring a target mark corresponding to the position information from the operation queue based on the position information of the message in the data queue;
processing the message according to the target mark; wherein said processing said message according to said target mark comprises: and if the target mark is an update mark and a receiver mark, deleting the message from the data queue, generating an update message according to the message, storing the update message into the data queue, and sending the update message to a receiver corresponding to the receiver mark.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811142226.XA CN110968431B (en) | 2018-09-28 | 2018-09-28 | Message processing method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811142226.XA CN110968431B (en) | 2018-09-28 | 2018-09-28 | Message processing method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110968431A CN110968431A (en) | 2020-04-07 |
| CN110968431B true CN110968431B (en) | 2023-06-20 |
Family
ID=70027035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811142226.XA Active CN110968431B (en) | 2018-09-28 | 2018-09-28 | Message processing method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110968431B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111970091A (en) * | 2020-08-06 | 2020-11-20 | 北京浪潮数据技术有限公司 | Communication method, system, equipment and computer readable storage medium |
| CN111988391A (en) * | 2020-08-14 | 2020-11-24 | 网联清算有限公司 | Message sending method and device |
| CN112134938B (en) * | 2020-09-09 | 2022-08-26 | 重庆易宠科技有限公司 | Message processing method, device, terminal and computer readable storage medium |
| CN112069438B (en) * | 2020-09-10 | 2023-12-22 | 北京五八信息技术有限公司 | Message merging method, message processing method and device, equipment and medium |
| CN112286973B (en) * | 2020-11-19 | 2022-09-30 | 每日互动股份有限公司 | Data message storage method and device, computer equipment and storage medium |
| CN113296976A (en) * | 2021-02-10 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Message processing method, message processing device, electronic equipment, storage medium and program product |
| CN112925642A (en) * | 2021-02-25 | 2021-06-08 | 百果园技术(新加坡)有限公司 | Delayed message processing method, device, equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102469033A (en) * | 2010-11-01 | 2012-05-23 | 阿里巴巴集团控股有限公司 | Message subscription system and message sending method |
| CN104731912A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Message transmission method and device for message middleware MQ |
| CN105242975A (en) * | 2015-08-27 | 2016-01-13 | 浪潮软件股份有限公司 | Message transmission method and message middleware |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN107133302A (en) * | 2017-04-28 | 2017-09-05 | 努比亚技术有限公司 | Realize method, system, terminal and the computer-readable recording medium of data consistency |
| CN108009027A (en) * | 2017-11-23 | 2018-05-08 | 北京百度网讯科技有限公司 | Implementation method, device, equipment and the storage medium of queue message uniformity |
| CN108170545A (en) * | 2018-01-08 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of method for message transmission and device based on message-oriented middleware |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8577976B2 (en) * | 2010-04-27 | 2013-11-05 | International Business Machines Corporation | Application of system level policy in message oriented middleware |
-
2018
- 2018-09-28 CN CN201811142226.XA patent/CN110968431B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102469033A (en) * | 2010-11-01 | 2012-05-23 | 阿里巴巴集团控股有限公司 | Message subscription system and message sending method |
| CN104731912A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Message transmission method and device for message middleware MQ |
| CN105242975A (en) * | 2015-08-27 | 2016-01-13 | 浪潮软件股份有限公司 | Message transmission method and message middleware |
| CN105471714A (en) * | 2015-12-09 | 2016-04-06 | 百度在线网络技术（北京）有限公司 | Message processing method and device |
| CN107133302A (en) * | 2017-04-28 | 2017-09-05 | 努比亚技术有限公司 | Realize method, system, terminal and the computer-readable recording medium of data consistency |
| CN108009027A (en) * | 2017-11-23 | 2018-05-08 | 北京百度网讯科技有限公司 | Implementation method, device, equipment and the storage medium of queue message uniformity |
| CN108170545A (en) * | 2018-01-08 | 2018-06-15 | 北京奇艺世纪科技有限公司 | A kind of method for message transmission and device based on message-oriented middleware |
Non-Patent Citations (4)
| Title |
|---|
| He Ming Yao Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong等.Machine learning based method of moments (ML-MoM).2017 IEEE International Symposium on Antennas and Propagation & USNC/URSI National Radio Science Meeting.2017,全文. * |
| 史冬冬.云队列：一个基于Hadoop的大规模消息基础平台.中国优秀硕士学位论文全文数据库 信息科技辑.2012,(第07期),全文. * |
| 汪添生;尉双梅;崔蔚;刘迪;何金陵;孙琦.支持MongoDB的事务管理方案研究.计算机系统应用.2017,第26卷(第07期),全文. * |
| 许静;牟艳;秦江龙.竞赛数据交换平台的研究与实现.微处理机.2012,第33卷(第06期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110968431A (en) | 2020-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110968431B (en) | Message processing method, device and equipment | |
| EP2998863B1 (en) | Converting a serial transaction schedule to a parallel transaction schedule | |
| CN101686209B (en) | Method and device for storing message in message retransmission system | |
| US9253133B2 (en) | Message thread identification and management | |
| US20040119740A1 (en) | Methods and apparatus for displaying and replying to electronic messages | |
| CN106844662B (en) | Method and device for recording reading state of instant message | |
| CN104618432B (en) | A kind of processing method and processing system that event sends and receives | |
| CN110968586A (en) | Distributed transaction processing method and device | |
| CN107395559B (en) | Data processing method and device based on redis | |
| US9992147B2 (en) | During an email migration, converting distribution list messages into single-instanced content within a collaboration platform group | |
| US20160350431A1 (en) | Locating previously communicated electronic messages | |
| CN110728558A (en) | Virtual article package sending method, device, equipment and storage medium | |
| CN101571811B (en) | Information transmission method and device thereof | |
| CN112118297B (en) | Control method, device, equipment and storage medium for delay message | |
| CN112804359A (en) | Method and apparatus for providing cross-chain messages | |
| CN112152872A (en) | Network sub-health detection method and device | |
| CN107563942B (en) | Logistics data batch processing method, logistics processing system and processing device | |
| CN109905459A (en) | A kind of data transmission method and device | |
| US20150365365A1 (en) | Method and apparatus for modifying message | |
| CN110830361B (en) | Mail data storage method and device | |
| CN108694219B (en) | Data processing method and device | |
| CN108632128A (en) | A kind of email tracking method and apparatus | |
| AU2018280855B2 (en) | Multi-standard message processing | |
| CN114721755A (en) | To-do transaction message processing method and device and storage medium | |
| CN107835124B (en) | Mail sending method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |