CN110008041B - Message processing method and device - Google Patents
Message processing method and device Download PDFInfo
- Publication number
- CN110008041B CN110008041B CN201910236790.6A CN201910236790A CN110008041B CN 110008041 B CN110008041 B CN 110008041B CN 201910236790 A CN201910236790 A CN 201910236790A CN 110008041 B CN110008041 B CN 110008041B
- Authority
- CN
- China
- Prior art keywords
- message
- processed
- server
- thread
- recorded data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a message processing method and a device, wherein the method comprises the following steps: a first server receives a message to be processed sent by a message queue middleware, wherein the message to be processed carries a user ID; the first server acquires a first thread distributed for the user ID by using the user ID, wherein the first thread is used for processing each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when the message in a database of a second server changes; the first server reads the message to be processed through the first thread and locks the message to be processed in a multi-thread mode; and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through the first thread.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and an apparatus for processing a message.
Background
Data inconsistency occurs in databases between different systems due to various uncertainties, such as server link timeouts of any one system. The details will be described below.
Firstly, a user sends a message A to an X system; second, X system receives message A from user, X system is in X of X system1Writing the message A into the database; thirdly, when x is monitored1The state of the A message in the database changes, wherein the state of the A message changes, such as the A message is read or updated, wherein the updated A message is the same as the A messageA message with the same content sent by a user, wherein the time stamps carried by the message with the same content are different, the message queue is used for recording the changed message A, and the changed message A is sent to the Y system; fourthly, after the Y system receives the changed A message, the Y system prepares the Y system in the Y system1The database is written with this changed a message. However, the Y system is in Y1The uncertain factor of network abnormality occurs in the process of writing the changed A message in the database, so that y1The changed A message is not successfully written into the database, and the changed A message is considered to belong to the unconsumed message and returns to the end of the message queue again.
The payment settlement system is taken as an example for explanation: the user sends a message of requesting payment to the payment settlement system, the payment settlement system receives the message of requesting payment sent by the user, and the message is x in the payment settlement system1Writing a payment request message in a database; when x is monitored1The method comprises the steps that a payment request message written in a database is updated, the message queue is used for recording the changed payment request message according to the principle of first-in first-out, the changed payment request message is sent to a payment authentication system, and the payment authentication system is prepared to write the changed payment request message in the database of the payment authentication system after receiving the changed payment request message. The changed payment request message is written in the payment authentication system, and the changed payment request message is not written in the database of the payment authentication system due to the uncertain factors of the network abnormity, is considered to belong to the unconsumed message at the moment, and returns to the end of the message queue again.
However, the inventor finds that in the process of implementing the invention, the prior art can not only exist Y of the Y system1X of database and X system1The problem of inconsistent messages in the database, and because the message queue stores the messages for consumption according to the principle of first-in first-out, other messages in the first-in message queue are consumed before the A message in the current message queue, and then the A message is consumed。
In summary, the prior art has problems of inconsistent messages of databases of different systems and messy message processing.
Disclosure of Invention
An object of the embodiments of the present invention is to provide a method and an apparatus for processing a message, so as to solve the problems in the prior art that messages of databases of different systems are inconsistent and the processed messages are messy. The specific technical scheme is as follows:
in a first aspect, the present invention provides a message processing method, including:
a first server receives a message to be processed sent by a message queue middleware, wherein the message to be processed carries a user ID;
the first server acquires a first thread distributed for the user ID by using the user ID, wherein the first thread is used for processing each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when the message in a database of a second server changes;
the first server reads the message to be processed through the first thread and locks the message to be processed in a multi-thread mode;
and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through the first thread.
Further, when it is determined that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through the first thread, including:
the first server acquires recorded data in a local cache;
and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the recorded data in the local cache, the first server sends a consumption success confirmation of the message to be processed through the first thread.
Further, when it is determined that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through the first thread, including:
the first server acquires recorded data in a local cache;
when the first server determines that the timestamp carried by the message to be processed is not earlier than the latest timestamp of the recorded data in the local cache, the first server acquires the recorded data in a data update table, wherein the data update table is used for recording a data table of an operation log of each database;
and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the recorded data in the data updating table, the first server sends a consumption success confirmation of the message to be processed through the first thread.
Further, after the first server obtains the recorded data in the data update table, the method further includes:
when the first server determines that the timestamp carried by the message to be processed is not earlier than the latest timestamp of the recorded data in the data updating table, writing the message to be processed into a database of the first server through the first server;
and when the first server determines that the message to be processed is successfully written into the database of the first server, the first server sends a consumption success confirmation of the message to be processed.
Further, the writing into the database of the first server includes:
and when determining that the message to be processed fails to be written into the database of the first server, the first server sends abandon confirmation consumption of the message to be processed.
In a second aspect, the present invention provides a message processing apparatus, applied to a first server, the message processing apparatus including:
the receiving module is used for receiving a message to be processed sent by the message queue middleware, and the message to be processed carries a user ID;
an obtaining module, configured to obtain, by using the user ID, a first thread allocated to the user ID, where the first thread is used to process each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when a message in a database of a second server changes;
the reading module is used for reading the message to be processed through the first thread and locking the message to be processed in a multi-thread mode;
and the first processing module is used for sending a message success confirmation message of the message to be processed through the first thread when the fact that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data is determined.
Further, the first processing module is configured to:
acquiring recorded data in a local cache;
and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the recorded data in the local cache, sending a consumption success confirmation of the message to be processed through the first thread.
Further, the first processing module is configured to:
acquiring recorded data in a local cache;
when the time stamp carried by the message to be processed is determined to be not earlier than the latest time stamp of the recorded data in the local cache, the recorded data in a data update table is obtained, and the data update table is used for recording a data table of an operation log of each database;
and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the recorded data in the data updating table, sending a consumption success confirmation of the message to be processed through the first thread.
Further, the apparatus further comprises:
the second processing module is used for writing the message to be processed into the database of the first server after the recorded data in the data update table is acquired and when the time stamp carried by the message to be processed is determined to be not earlier than the latest time stamp of the recorded data in the data update table;
and the sending module is used for sending the consumption success confirmation of the message to be processed when the message to be processed is successfully written into the database of the first server.
Further, the sending module is further configured to:
and when determining that the to-be-processed message fails to be written into the database of the first server, sending abandon confirmation consumption of the to-be-processed message.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of the first aspect when executing the program stored in the memory.
In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon instructions which, when run on a computer, cause the computer to perform the method of any of the first aspects described above.
In a fifth aspect, the present invention also provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any of the first aspects described above.
According to the message processing method and device provided by the embodiment of the invention, a first server receives a message to be processed sent by a middleware, acquires a first thread allocated for a user ID by using the user ID carried by the message to be processed, reads the message to be processed through the first thread, and locks the message to be processed in a multi-thread manner; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, sending a message success confirmation message of the message to be processed through the first thread.
Therefore, the same user uses the same first thread to process the messages to be processed, and the first thread processes the messages of the user ID of the same message to be processed through the multi-thread lock, so that the problem that the same message to be processed is used by multiple threads and the processed messages are disordered is avoided; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, the message to be processed is not written into the database of the first server, but the message success confirmation message of the message to be processed is sent through the first thread. Compared with the prior art, the method has the advantages that the processing result of the message to be processed is prevented from covering the consumed result of the recorded data of the latest timestamp, and the message is prevented from being processed in disorder; in addition, the recorded data acquired by the first server can keep the consistency of the data of the first server and the second server.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a schematic flowchart of a message processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a process of processing a message to be processed according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a message processing apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a message processing system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
Aiming at the problems of inconsistent information of databases of different systems and messy information processing in the prior art, the embodiment of the invention provides an information processing method and a device, which adopt the following steps to complete the information processing:
the method comprises the steps that a first server receives a message to be processed sent by a middleware, a first thread allocated for a user ID is obtained by the user ID of the message to be processed, the message to be processed is read through the first thread, and multithreading locking is carried out on the message to be processed; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, sending a message success confirmation message of the message to be processed through the first thread.
Therefore, for the same user, the message to be processed is processed through the same first thread, and the first thread processes each message of the user ID of the same message to be processed through the multi-thread lock, so that the problem that the same message to be processed is used by multiple threads and the message processing is messy is avoided; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, the message to be processed is not written into the database of the first server, but the message success confirmation message of the message to be processed is sent through the first thread. Compared with the prior art, the method has the advantages that the processing result of the message to be processed is prevented from covering the consumed result of the recorded data of the latest timestamp, and the message is prevented from being processed in disorder; in addition, the recorded data acquired by the first server can keep the consistency of the data of the first server and the second server.
First, a message processing method provided in an embodiment of the present invention is described below.
Referring to fig. 1, fig. 1 is a schematic flowchart of a message processing method according to an embodiment of the present invention. The message processing method provided by the embodiment of the invention can comprise the following steps:
in step 110, the first server receives a to-be-processed message sent by the message queue middleware, where the to-be-processed message carries a user ID.
The message queue middleware is used for recording the change message when the message in the database of the second server changes. This Message Queue middleware may be Message Queue (MQ) middleware. This MQ middleware may wait for messages to be processed on a "first-in-first-out" basis. Currently, RabbitMQ, rockmq, ActiveMQ, Kafka, ZeroMQ, MetaMQ, etc. are included through more message queue middleware. This achieves final data consistency. The messages to be processed are reliably transmitted through the MQ middleware, and the messages which are abandoned and confirmed to be consumed are retried on the basis of ensuring the idempotency until the messages are successful.
The database of the second server in the embodiment of the present invention may be the same as the database of the first server, or the database of the second server may be different from the database of the first server, and the database of the second server and the database of the first server are not limited. In addition, the number of the databases is not limited, and the number of the databases can be set according to the requirements of users. These databases may also store part of the same data, which is provided for different services. By way of example, the database of the second server may be, but is not limited to: MySQL database, the database of the first server may be but is not limited to: hbase database. The second server may be a server in a server other than the first server. The "second" in the second server and the "first" in the first server are used to distinguish the servers, and are not limited in sequence.
The first server may be, but is not limited to, a server that obtains a change message after the change of the message occurs in the database of the second server, and is about to write the change message. The server comprises a database of a first server, the database of the first server is used for recording the position of the change message, and the change message is written into the database of the first server for keeping consistency with data in the database of a second server.
The pending message may be a change message received by the message queue middleware, and the change message is sent when a message in the database of the second server changes. When the second server monitors that the message in the database of the second server changes, the implementation process of sending the change message may be as follows, but is not limited to this: and under the condition that the user side interacting with the second server sends the message to the second server, when the second server receives the message, the second server writes the message into a database of the second server. And when the second server monitors that the information in the database of the second server changes, the second server sends the change information to the information queue middleware so that the information queue middleware can record the change information in time. Therefore, when the message is changed in the database of the second server, the change message can be determined in time, and the change message is recorded by the message queue middleware.
And when the database of the second server is the MySQL database, the second server monitors that
Sending the change message when the message changes in the database of the second server further comprises: and the second server sends a change message when monitoring that the message in the binlog log of the MySQL database of the second server changes. Where the binlog log is used to record all statements that have updated data or have potentially updated data.
The number of the messages to be processed is not limited. Moreover, the second server may refer to a server other than the first server, and as long as the first server and the second server are capable of performing communication interaction, both of them belong to the protection scope of the embodiment of the present invention. Illustratively, the first server may be, but is not limited to being, a server of an authentication system. Therefore, the server back end of the authentication system synchronizes data, and the data seen by a third party is ensured to have consistency.
Step 110 specifically, but not exclusively, includes: and after the change message is sent to the message queue middleware when the second server monitors that the message in the database of the second server changes, the message queue middleware receives the change message sent by the message queue middleware.
After step 110, the method further comprises: the first server processes the message and feeds back the result of whether the message is confirmed to be consumed. For example, the message to be processed is successfully written into the database of the first server, and the first server feeds back the consumption confirmation message; for another example, if the pending message fails to be written into the database of the first server, the first server returns a discard confirmation consumption. Therefore, the repeated consumption of the same message to be processed can be avoided by feeding back whether the result of the message consumption is confirmed or not.
In step 110, when the user side interacting with the second server is used, an identifier, i.e. a user ID, of the user identity using the user side is identified, where the user ID may be, but is not limited to, a user account. Therefore, the message sent by the user side can be mastered, so that the message can be processed in the later period. For the step, the message to be processed also contains the user ID of the message to be processed, and through the user ID of the message to be processed, it is convenient to allocate a thread to the user ID of the message to be processed, and the thread processes each message generated by the user ID of the message to be processed, and the thread may be referred to as a first thread.
When different threads process messages with the same user ID, message confusion occurs, which is specifically described as follows:
it is assumed that the execution flow required by the first server is that the first thread processes the message first, then the result of the first thread is used as the input of the second thread, and the second thread processes the result to obtain the final result, so that the final result is the result of the second thread. However, there are thread factors, the first thread and the second thread process the message at the same time, the second thread obtains the result first, and then the first thread obtains the result, so that the result of the first thread covers the result of the second thread, the final result is the result of the first thread, and the final result is an error.
Specific application examples are as follows:
the execution process required by the first server is assumed to be that the first thread receives the payment request message first to obtain the result of whether the request of the payment request message is successful; then, the result of whether the request for requesting the payment message is successful is used as the input of the second thread; the second thread follows the result of whether the request of the payment request message is successful or not and determines whether to send a notification message or not; wherein, the successful request of the payment request message sends a notification message of the successful request of the payment request message, the unsuccessful request of the payment request message does not send the notification message, and thus the result of the second thread comprises: whether to send a notification message. However, the first thread and the second thread may process the request payment message at the same time due to the fact that the thread factor exists, the second thread obtains a result that the notification message is not sent first, and then the first thread receives the request payment message to obtain a result that the request of the request payment message is successful, so that the result that the request of the request payment message is successful may cover the result that the notification message is not sent, and the result is wrong.
Therefore, in order to solve the problem that the processing of messages with the same user ID is messy due to different threads, in the embodiment of the present invention, one thread is allocated to one user ID, but one thread can process messages with no user ID. This eliminates the need for threads to determine the order of processing, and controls each thread by using a multi-threaded lock.
For the first thread, the first thread may process each message generated by the user ID of different messages to be processed, but all messages of the user ID of one message to be processed are processed by the same first thread, so as to avoid messiness in processing messages caused by different threads processing the user ID of the same message to be processed at different speeds.
The obtaining manner of the first thread allocated to the user ID of the message to be processed in this step 120 is:
in one implementation mode, according to the user ID of the message to be processed, finding out whether the thread of the user ID of the message to be processed exists in the distributed threads; if the thread of the user ID of the message to be processed is found from the distributed threads, the thread of the user ID of the message to be processed is obtained and is used as a first thread; and if the thread of the user ID of the message to be processed is not found from the assigned threads, a first thread is assigned to the user ID of the message to be processed. Wherein the assigned threads are used for recording the user ID of processing the message in each assigned thread.
And step 130, the first server reads the message to be processed through the first thread and locks the message to be processed in a multi-thread mode. This prevents multiple threads from simultaneously processing the same data in the database of the first server through multi-threaded locking.
In this step 130, when the database of the first server is the Hbase database, the same column of data in the Hbase database of the first server is prevented from being processed simultaneously by multiple threads through the multiple thread locking.
The recorded data may include only the latest timestamp, or may include other recorded data and the latest timestamp. The recorded data may be, but is not limited to, data in the first server, data in the second server, or data independent of the first server and the second server, respectively. The recorded data may be recorded in a data table or a database, and is not limited herein. The data table may be a single data table, or a plurality of data tables may be established for each different database. The following description will be made with reference to the data table as a data update table.
In order to obtain the recorded data, the method of the embodiment of the present invention further includes:
and acquiring data in a data update table as recorded data from a server independent of the first server and the second server, wherein the data update table is used for recording a data table of operation logs of each database of the first server and the second server. The recorded data in the data update tables are created by acquiring operation logs from the databases of the first server and the second server.
This step may include, but is not limited to: sending a request for acquiring a data update table to a server independent of the first server and the second server; and receiving a data update table fed back by a server independent from the first server and the second server. Therefore, the data update table can be acquired from a server independent of the first server and the second server, and the storage data of the first server and the second server can be reduced.
In order to obtain the recorded data, the method of the embodiment of the present invention further includes:
and acquiring the data in the data updating table from the second server as the recorded data.
This step may include, but is not limited to: sending a request for obtaining a data update table to a second server; and receiving a data update table fed back by the second server. In this way, the data update table can be obtained from the second server, and all data in the data update table can be directly read.
In order to obtain the recorded data, the method of the embodiment of the present invention further includes:
and acquiring the data from the local cache of the first server as the recorded data. Therefore, the local cache can store partial data in the data updating table, and the data volume of directly reading the data updating table can be reduced.
In order to process the message to be processed, referring to fig. 2, this step 140 may be implemented by at least one of the following implementations:
in one implementation, in order to solve the problem that if a pending message is retried for multiple times, the sequence of the pending message cannot be guaranteed, and the old data overwrites the new data, step 141, the first server obtains the recorded data in the local cache. The local cache temporarily stores the latest timestamp of the recorded data, so that the execution time of the first server can be improved, and the Input/Output (I/O) delay generated by accessing the database can be reduced.
In this step 141, since the local cache records part of the data in the data update table, it may not be determined at one time that the timestamp carried by the pending message is earlier than the latest timestamp of the recorded data in the local cache.
Step 143, sending a consumption success confirmation of the pending message through the first thread. Therefore, the message to be processed is processed directly through the local cache, and the data volume of directly searching data from the data updating table is reduced.
In another implementation manner, in step 144, the first server obtains the recorded data in the data update table, where the data update table is used for recording the data table of the operation log of each database. Therefore, recorded data later than the message to be processed is not recorded in the local cache, the recorded data in the data updating table can be searched, and the searched data is more comprehensive.
the first server sends a confirmation of successful consumption of the pending message via the first thread, step 146.
This step 148 the step of the first server determining whether the pending message was successfully written to the database of the first server further comprises: receiving a processing result of successful writing, and determining that the message to be processed is successfully written into the database of the first server; and if the successfully written processing result is not received within the preset time, determining that the message to be processed fails to be written into the database of the first server, wherein the preset time can be set according to the requirements of users. Therefore, the first server can determine whether the message to be processed is successfully written into the database of the first server or not through the processing result.
Step 149, send a consumption success confirmation of the pending message.
In the embodiment of the invention, the same user processes the messages to be processed through the same first thread, and the first thread processes the messages of the user ID of the same message to be processed through the multi-thread lock, so that the problem that the same message to be processed is used by multiple threads and the processed messages are disordered is avoided; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, the message to be processed is not written into the database of the first server, but the message success confirmation message of the message to be processed is sent through the first thread. Compared with the prior art, the method has the advantages that the processing result of the message to be processed is prevented from covering the consumed result of the recorded data of the latest timestamp, and the message is prevented from being processed in disorder; in addition, the recorded data acquired by the first server can keep the consistency of the data of the first server and the second server. Therefore, the combination of the latest timestamp of the recorded data and the data update table is introduced, and the final consistency of the data is ensured by carrying out local verification on the message middleware and the first server.
The specific application scenario of the embodiment of the present invention is described with a first server in a payment authentication system as an example:
first, taking the payment settlement system as an example for explanation, the related art has the changed payment request message that is not consumed, and the result of the consumption at this time can override the updated result already existing in the updated changed payment request message, which can cause the problem of multiple payment requests: if there is an update of the changed solicited payment message, then the updated changed solicited payment message is added to the message queue after the changed solicited payment message. However, the modified request-for-payment message is not consumed, and the modified request-for-payment message is returned to the end of the message queue. The updated changed payment request message stores the message for consumption according to the principle of first-in first-out, the message has an updated result, and the changed payment request message which is not consumed before is executed according to the sequence of the message in the message queue, so that the updated result of the updated changed payment request message which is not consumed can be covered by the consumed result, and multiple times of payment requests can be caused.
Secondly, in order to solve the above problem, the embodiment of the present invention may adopt the following implementation processes from the first step to the seventh step, so that the latest result is not covered by the result obtained by the last payment request message in the queue, and the processing message is not disordered, and even multiple payments requests are not caused.
Firstly, a user sends a message of requesting payment to a payment settlement system;
second, the payment settlement system receives the payment request message sent by the user, and the payment settlement system is in the x1Writing a payment request message in a database;
thirdly, when x is monitored1The payment request message written in the database is updated, the message queue records the updated payment request message according to the principle of first-in first-out, and the updated payment request message is sent to a first server in the payment authentication system;
fourthly, the first server receives the updated payment request message sent by the message queue middleware, and the payment request message carries the user ID;
fifthly, the first server acquires a first thread distributed for the user ID by using the user ID;
sixthly, the first server reads the updated payment request message through the first thread and carries out multithread locking on the updated payment request message;
and seventhly, the first server sends a message success confirmation message of the message to be processed through the first thread when the fact that the timestamp carried by the updated request payment message is earlier than the latest timestamp of the request payment message in the acquired recorded data is determined. The time stamp carried by the updated request payment message is earlier than the latest time stamp of the request payment message in the acquired recorded data, which indicates that the updated request payment message is an expired message, the request payment message updated with the updated request payment message already exists, which is referred to as the latest request payment message for convenience of description, and the latest request payment message is consumed and has the latest result.
It follows that this updated requested payment message was not previously successfully written in the payment authentication system's database, i.e. this updated requested payment message would be considered to belong to a message that was not consumed, being returned to the end of the message queue. The latest payment request message and the latest payment request message returned to the queue are stored according to the principle of first-in first-out, the latest payment request message is consumed and has an updated result; and reading the last payment request message of the queue, wherein when the timestamp carried by the last payment request message of the queue is earlier than the latest timestamp of the recorded data, the last payment request message of the queue is not written into the database of the first server, namely, the last payment request message of the queue is not consumed again, and the previous payment request message which is not consumed, so that the latest result is not covered by the result obtained by the last payment request message of the queue, and the message processing is not disordered.
The following description is continued on a message processing apparatus according to an embodiment of the present invention.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a message processing apparatus according to an embodiment of the present invention. An embodiment of the present invention provides a message processing apparatus, which is applied to a first server, and the message processing apparatus includes:
a receiving module 21, configured to receive a to-be-processed message sent by a message queue middleware, where the to-be-processed message carries a user ID;
an obtaining module 22, configured to obtain, by using the user ID, a first thread allocated to the user ID, where the first thread is used to process each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when a message in a database of a second server changes;
a reading module 23, configured to read the to-be-processed message through the first thread, and lock the to-be-processed message in a multi-thread manner;
the first processing module 24 is configured to send a message success confirmation message of the message to be processed through the first thread when it is determined that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data.
In the embodiment of the invention, the same user processes the messages to be processed through the same first thread, and the first thread processes the messages of the user ID of the same message to be processed through the multi-thread lock, so that the problem that the same message to be processed is used by multiple threads and the processed messages are disordered is avoided; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, the message to be processed is not written into the database of the first server, but the message success confirmation message of the message to be processed is sent through the first thread. Compared with the prior art, the method and the device avoid the processing result of the message to be processed from covering the consumed result of the recorded data of the latest timestamp and avoid mess of the processed message.
In a possible implementation manner, the first processing module is configured to:
acquiring recorded data in a local cache;
and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the recorded data in the local cache, sending a consumption success confirmation of the message to be processed through the first thread.
In a possible implementation manner, the first processing module is configured to:
acquiring recorded data in a local cache;
when the time stamp carried by the message to be processed is determined to be not earlier than the latest time stamp of the recorded data in the local cache, the recorded data in a data update table is obtained, and the data update table is used for recording a data table of an operation log of each database;
and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the recorded data in the data updating table, sending a consumption success confirmation of the message to be processed through the first thread.
In one possible implementation, the apparatus further includes:
the second processing module is used for writing the message to be processed into the database of the first server after the recorded data in the data update table is acquired and when the time stamp carried by the message to be processed is determined to be not earlier than the latest time stamp of the recorded data in the data update table;
and the sending module is used for sending the consumption success confirmation of the message to be processed when the message to be processed is successfully written into the database of the first server.
In a possible implementation manner, the sending module is further configured to:
and when determining that the to-be-processed message fails to be written into the database of the first server, sending abandon confirmation consumption of the to-be-processed message.
In the embodiment of the invention, the same user processes the messages to be processed through the same first thread, and the first thread processes the messages of the user ID of the same message to be processed through the multi-thread lock, so that the problem that the same message to be processed is used by multiple threads and the processed messages are disordered is avoided; and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the acquired recorded data, the message to be processed is not written into the database of the first server, but the message success confirmation message of the message to be processed is sent through the first thread. Compared with the prior art, the method has the advantages that the processing result of the message to be processed is prevented from covering the consumed result of the recorded data of the latest timestamp, and the message is prevented from being processed in disorder; in addition, the recorded data acquired by the first server can keep the consistency of the data of the first server and the second server. Therefore, the combination of the latest timestamp of the recorded data and the data update table is introduced, and the final consistency of the data is ensured by carrying out local verification on the message middleware and the first server.
Referring to fig. 4, an embodiment of the present invention further provides a data transmission system, where the data transmission system includes:
assuming that the number of the second servers is one, the second server is called a second server, the databases in the second server are a MySQL database 31, a first server 33 of the authentication system, and a database in the first server is an Hbase database 35; the message queue middleware is MQ middleware 32; the recorded data is acquired from a server independent of the first server and the second server, and the server independent of the first server and the second server is referred to as a third server.
The user side interacting with the second server sends a message to the second server;
the second server receives a message sent by a user side interacting with the second server, writes the message into a database of the second server, and sends a change message to the message queue middleware when monitoring that the message in the MySQL database of the second server changes;
the message queue middleware records the change message and sends the change message to the first server;
the method comprises the steps that a first server receives a message to be processed sent by a message queue middleware, wherein the message to be processed carries a user ID; the first server acquires a first thread distributed for the user ID by using the user ID, wherein the first thread is used for processing each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when the message changes in a database of the second server; the first server reads the message to be processed through the first thread and carries out multithread locking on the message to be processed; when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through a first thread; when it is determined that the timestamp carried by the message to be processed is not earlier than the latest timestamp of the recorded data in the data update table 34, the first server writes the message to be processed into the Hbase database 35 of the first server; the first server determines that the message to be processed is successfully written into the Hbase database 35 of the first server, and sends a consumption success confirmation of the message to be processed.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The embodiment of the present invention further provides an electronic device, which includes a processor 41, a communication interface 42, a memory 43 and a communication bus 44, wherein the processor 41, the communication interface 42, and the memory 43 complete mutual communication through the communication bus 44,
a memory 43 for storing a computer program;
the processor 41, when executing the program stored in the memory 43, implements the following steps:
a first server receives a message to be processed sent by a message queue middleware, wherein the message to be processed carries a user ID;
the first server acquires a first thread distributed for the user ID by using the user ID, wherein the first thread is used for processing each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when the message in a database of a second server changes;
the first server reads the message to be processed through the first thread and locks the message to be processed in a multi-thread mode;
and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through the first thread.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For convenience, only one thick line is used in the figures, but there is not only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
The method provided by the embodiment of the invention can be applied to electronic equipment. Further, the electronic device may be: desktop computers, laptop computers, intelligent mobile terminals, servers, and the like. Without limitation, any electronic device that can implement the present invention is within the scope of the present invention.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, and when the instructions are executed on a computer, the instructions cause the computer to execute the message processing method described in any of the above embodiments.
In yet another embodiment, the present invention further provides a computer program product containing instructions which, when run on a computer, cause the computer to perform the message processing method described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optics, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.), the computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more integrated servers, data centers, etc., the available medium may be magnetic medium (e.g., floppy disk, hard disk, magnetic tape), optical medium (e.g., high density Digital Video Disc, DVD for short), or a semiconductor medium (such as a Solid State Disk (SSD for short)), etc.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus/electronic device/storage medium embodiment/computer program product containing instructions, the description is relatively simple as it is substantially similar to the method embodiment, and reference may be made to some descriptions of the method embodiment for relevant points.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (11)
1. A message processing method, comprising:
a first server receives a message to be processed sent by a message queue middleware, wherein the message to be processed carries a user ID;
the first server acquires a first thread distributed for the user ID by using the user ID, wherein the first thread is used for processing each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when the message in a database of a second server changes;
the first server reads the message to be processed through the first thread and locks the message to be processed in a multi-thread mode;
when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, the first server sends a message success confirmation message of the message to be processed through the first thread, wherein the message success confirmation message is as follows: successfully identifying whether a message of the pending message has been consumed.
2. The method as claimed in claim 1, wherein the sending, by the first thread, the message success confirmation message of the message to be processed when it is determined that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, comprises:
the first server acquires recorded data in a local cache;
and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the recorded data in the local cache, the first server sends a consumption success confirmation of the message to be processed through the first thread.
3. The method as claimed in claim 1, wherein the sending, by the first thread, the message success confirmation message of the message to be processed when it is determined that the timestamp carried by the message to be processed is earlier than the latest timestamp of the acquired recorded data, comprises:
the first server acquires recorded data in a local cache;
when the first server determines that the timestamp carried by the message to be processed is not earlier than the latest timestamp of the recorded data in the local cache, the first server acquires the recorded data in a data update table, wherein the data update table is used for recording a data table of an operation log of each database;
and when the first server determines that the timestamp carried by the message to be processed is earlier than the latest timestamp of the recorded data in the data updating table, the first server sends a consumption success confirmation of the message to be processed through the first thread.
4. The method of claim 3, wherein after the first server obtains the recorded data in the data update table, the method further comprises:
when the first server determines that the timestamp carried by the message to be processed is not earlier than the latest timestamp of the recorded data in the data updating table, writing the message to be processed into a database of the first server through the first server;
and when the first server determines that the message to be processed is successfully written into the database of the first server, the first server sends a consumption success confirmation of the message to be processed.
5. The method of claim 4, wherein the writing to the database of the first server comprises:
and when determining that the message to be processed fails to be written into the database of the first server, the first server sends abandon confirmation consumption of the message to be processed.
6. A message processing apparatus, applied to a first server, the apparatus comprising:
the receiving module is used for receiving a message to be processed sent by the message queue middleware, and the message to be processed carries a user ID;
an obtaining module, configured to obtain, by using the user ID, a first thread allocated to the user ID, where the first thread is used to process each message generated by the user ID, the message to be processed is a change message received by the message queue middleware, and the change message is sent when a message in a database of a second server changes;
the reading module is used for reading the message to be processed through the first thread and locking the message to be processed in a multi-thread mode;
a first processing module, configured to send a message success confirmation message of the message to be processed through the first thread when it is determined that a timestamp carried by the message to be processed is earlier than a latest timestamp of the acquired recorded data, where the message success confirmation message is: successfully identifying whether a message of the pending message has been consumed.
7. The apparatus of claim 6, wherein the first processing module is to:
acquiring recorded data in a local cache;
and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the recorded data in the local cache, sending a consumption success confirmation of the message to be processed through the first thread.
8. The apparatus of claim 6, wherein the first processing module is to:
acquiring recorded data in a local cache;
when the time stamp carried by the message to be processed is determined to be not earlier than the latest time stamp of the recorded data in the local cache, the recorded data in a data update table is obtained, and the data update table is used for recording a data table of an operation log of each database;
and when the time stamp carried by the message to be processed is determined to be earlier than the latest time stamp of the recorded data in the data updating table, sending a consumption success confirmation of the message to be processed through the first thread.
9. The apparatus of claim 8, wherein the apparatus further comprises:
the second processing module is used for writing the message to be processed into the database of the first server after the recorded data in the data update table is acquired and when the time stamp carried by the message to be processed is determined to be not earlier than the latest time stamp of the recorded data in the data update table;
and the sending module is used for sending the consumption success confirmation of the message to be processed when the message to be processed is successfully written into the database of the first server.
10. The apparatus of claim 9, wherein the sending module is further configured to:
and when determining that the to-be-processed message fails to be written into the database of the first server, sending abandon confirmation consumption of the to-be-processed message.
11. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
the memory is used for storing a computer program;
the processor, when executing the program stored in the memory, implementing the method steps of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910236790.6A CN110008041B (en) | 2019-03-27 | 2019-03-27 | Message processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910236790.6A CN110008041B (en) | 2019-03-27 | 2019-03-27 | Message processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110008041A CN110008041A (en) | 2019-07-12 |
| CN110008041B true CN110008041B (en) | 2021-03-19 |
Family
ID=67168355
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910236790.6A Active CN110008041B (en) | 2019-03-27 | 2019-03-27 | Message processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110008041B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110647409B (en) * | 2019-09-23 | 2023-05-12 | 深圳市网心科技有限公司 | Message writing method, electronic device, system and medium |
| CN112711436B (en) * | 2019-10-25 | 2023-09-26 | 北京奇艺世纪科技有限公司 | Message processing method and device |
| CN111708618A (en) * | 2020-06-12 | 2020-09-25 | 北京思特奇信息技术股份有限公司 | Processing method and device based on Java multithreading |
| CN113626217A (en) * | 2021-07-28 | 2021-11-09 | 北京达佳互联信息技术有限公司 | Asynchronous message processing method and device, electronic equipment and storage medium |
| CN113742112B (en) * | 2021-09-15 | 2024-04-16 | 武汉联影智融医疗科技有限公司 | Electrocardiogram image generation method, system and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1917504A (en) * | 2005-08-20 | 2007-02-21 | 中兴通讯股份有限公司 | Method for preventing deadlock caused by accessing shared data of resources |
| CN101996099A (en) * | 2010-11-17 | 2011-03-30 | 山东中创软件工程股份有限公司 | Method and system for processing information |
| CN106776312A (en) * | 2016-12-12 | 2017-05-31 | 北京蓝海讯通科技股份有限公司 | Method, device, computing device and pressure testing system that cloud pressure testing data is distributed |
| CN107957903A (en) * | 2017-11-13 | 2018-04-24 | 中国平安财产保险股份有限公司 | Asynchronous task scheduling method, server and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9998416B2 (en) * | 2015-03-25 | 2018-06-12 | International Business Machines Corporation | Selecting subsets of participants in electronic message threads |
-
2019
- 2019-03-27 CN CN201910236790.6A patent/CN110008041B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1917504A (en) * | 2005-08-20 | 2007-02-21 | 中兴通讯股份有限公司 | Method for preventing deadlock caused by accessing shared data of resources |
| CN101996099A (en) * | 2010-11-17 | 2011-03-30 | 山东中创软件工程股份有限公司 | Method and system for processing information |
| CN106776312A (en) * | 2016-12-12 | 2017-05-31 | 北京蓝海讯通科技股份有限公司 | Method, device, computing device and pressure testing system that cloud pressure testing data is distributed |
| CN107957903A (en) * | 2017-11-13 | 2018-04-24 | 中国平安财产保险股份有限公司 | Asynchronous task scheduling method, server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110008041A (en) | 2019-07-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110008041B (en) | Message processing method and device | |
| WO2021180025A1 (en) | Message processing method and apparatus, electronic device and medium | |
| US9171019B1 (en) | Distributed lock service with external lock information database | |
| CN109309712B (en) | Data transmission method based on interface asynchronous call, server and storage medium | |
| US20060074847A1 (en) | Method and system for data processing with data replication for the same | |
| CN107341062B (en) | Data pushing method, device, equipment and storage medium | |
| US11233874B2 (en) | Ordinary write in distributed system maintaining data storage integrity | |
| CN107153644B (en) | Data synchronization method and device | |
| WO2019037617A1 (en) | Data transaction processing method, device, and electronic device | |
| US9514170B1 (en) | Priority queue using two differently-indexed single-index tables | |
| CN110737388A (en) | Data pre-reading method, client, server and file system | |
| US20210072903A1 (en) | Future write in distributed system maintaining data storage integrity | |
| CN108512930B (en) | Shared file management method, device, server and storage medium | |
| CN113687781A (en) | Method, device, equipment and medium for pulling up thermal data | |
| US9069632B2 (en) | Message processing | |
| CN113342507B (en) | Distributed lock service realization method and device and computer equipment | |
| CN112148206A (en) | Data reading and writing method and device, electronic equipment and medium | |
| CN112363762B (en) | Fusion command processing method, system, device and medium | |
| CN112865927B (en) | Message delivery verification method, device, computer equipment and storage medium | |
| CN111309693A (en) | Data synchronization method, device and system, electronic equipment and storage medium | |
| CN111078643B (en) | Method and device for deleting files in batch and electronic equipment | |
| CN110955460A (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN112860746B (en) | Cache reduction-based method, equipment and system | |
| US11288140B2 (en) | Point in time blob restore | |
| CN113726885A (en) | Method and device for adjusting flow quota |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |