CN116991603A

CN116991603A - Message synchronization method, device and computer readable storage medium

Info

Publication number: CN116991603A
Application number: CN202310961279.9A
Authority: CN
Inventors: 刘良; 刘国双; 唐宽
Original assignee: China Merchants Bank Co Ltd
Current assignee: China Merchants Bank Co Ltd
Priority date: 2023-07-31
Filing date: 2023-07-31
Publication date: 2023-11-03

Abstract

The invention discloses a message synchronization method, equipment and a computer readable storage medium, belonging to the technical field of message queues. The method comprises the following steps: the compensation node acquires a corresponding production record table and consumption record table from the database; screening production records meeting a first specific state from the production record table, and screening consumption records meeting a second specific state from the consumption record table; and sending the abnormal message corresponding to the production record and the consumption record to the message queue, and forwarding the abnormal message to the consumption node by the message queue. The present invention ensures consistency of data within a system by aiming at providing a unified and complete solution.

Description

Message synchronization method, device and computer readable storage medium

Technical Field

The present invention relates to the field of message queue technology, and in particular, to a method, an apparatus, and a computer readable storage medium for synchronizing messages.

Background

And a Message Queue (MQ) is used in the service system, so that the effects of asynchronous communication, decoupling of the system, flow peak clipping, improvement of the scalability and reliability of the system and the like can be realized. For example, in an intermediate business pricing management system for a financial institution, the start-up approval step of "submit pricing sheet interface" is time consuming and the system introduces a message queue to optimize the performance of the interface. So that after the pricing sheet is created, an approval message of the pricing sheet is sent to the message queue, which then automatically sends the approval message to the consuming node. Therefore, the system only needs to pay attention to the sending operation of the production node, and the system can return immediately after the sending operation is finished, so that an asynchronous starting approval instance is realized, and the overall performance of the interface is improved.

In the process of using the message queue, due to the reasons of internal abnormality of middleware of the message queue, downtime of service of consuming nodes and the like, abnormal conditions such as message loss, repeated consumption of the message and the like inevitably occur, and data inconsistency of a service system is further caused.

In the related art, after detecting that the data is inconsistent, the service system performs a message retry, and if the retry still fails, the service system performs manual intervention. However, the processing ideas and in particular the landing codes of different developers for the above cases are not uniform, i.e. lack a uniform and complete solution.

The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.

Disclosure of Invention

The invention mainly aims to provide a message synchronization method, which aims to solve the technical problem that a unified and complete solution is lacking under the condition of inconsistent data of a service system.

In order to achieve the above object, the present invention provides a message synchronization method, which is characterized in that the method is applied to a business approval system, the business approval system is provided with a message queue, a database, a production node, a consumption node and a compensation node, and the message synchronization method comprises the following steps:

The compensation node acquires a corresponding production record table and consumption record table from the database;

screening production records meeting a first specific state from the production record table, and screening consumption records meeting a second specific state from the consumption record table;

and sending the abnormal message corresponding to the production record and the consumption record to the message queue, and forwarding the abnormal message to the consumption node by the message queue.

Optionally, the step of screening the production record table for the production record satisfying the first specific state includes:

determining the state types and state holding time length of all production records in the production record table;

screening the production record with the status type of failed transmission from the production record list; and/or

Screening out the production record of which the state type is in transmission and the state holding time exceeds the longest waiting time from the production record table;

the step of screening the consumption record table for the consumption record meeting the second specific state comprises the following steps:

determining the state types of all consumption records in the consumption record table;

and screening the consumption record with the status type of consumption failure from the consumption record table.

Optionally, before the step of the compensation node obtaining the corresponding production record table and consumption record table from the database, the method includes:

after receiving a pricing bill uploaded by a user, the production node sends a corresponding writing instruction to the database, and the database inserts a corresponding production record into a production record table based on the writing instruction, wherein the state type of the production record is in the sending process;

the production node sends the pricing bill to a message queue and receives a determined signal value returned by the message queue;

if the confirmation signal value is a first signal value, a first change instruction is sent to the database, and the database changes the state type of the production record into successful sending based on the first change instruction;

and if the confirmation signal value is a second signal value, sending a second change instruction to the database, and changing the state type of the production record into a sending failure by the database based on the second change instruction.

Optionally, before the step of the production node sending the production record to a message queue and receiving the determined signal value returned by the message queue, the method includes:

The method comprises the steps that a production node obtains the available capacity of a message queue, and the state type in a production record table is the sending number of consumption records in sending;

calculating an overflow risk value according to the available capacity and the transmission quantity;

and if the overflow risk value is lower than a preset risk value, executing the steps that the production node sends the pricing sheet to a message queue and receives a determined signal value returned by the message queue.

Optionally, after receiving the pricing sheet uploaded by the user, the production node sends a corresponding writing instruction to the database, and the database inserts a corresponding production record into the production record table based on the writing instruction, where the status type of the production record is that after the step in sending, the method includes:

determining a production record element of the production record;

acquiring consumption record elements of all consumption records in the consumption record table;

and if the consumption pricing element which is the same as the production pricing element exists, sending a third changing instruction to the database, and changing the state type of the production record into a state type which does not need to be sent based on the third changing instruction by the database.

After receiving the pricing bill sent by the message queue, the consumption node sends a corresponding writing instruction to a database, and the database inserts a corresponding consumption record into a consumption record table based on the writing instruction;

the database judges whether the primary key of the consumption record has conflict or not;

if the conflict exists, setting the state type of the consumption record as repeated consumption;

and if no conflict exists, setting the state type of the consumption record as consumption.

Optionally, after the step of setting the status type of the consumption record to be in consumption if there is no conflict, the method includes:

the consumption node screens out the consumption record of which the state type is in consumption, and executes corresponding business logic on a pricing bill corresponding to the consumption record;

if the service logic is executed successfully, a fourth change instruction is sent to the database, and the database changes the state type of the consumption record into consumption success based on the fourth change instruction and returns a corresponding confirmation signal value to the message queue;

and if the execution service logic fails, a fifth change instruction is sent to the database, and the database changes the state type of the consumption record into consumption failure based on the fifth change instruction and returns a corresponding confirmation signal value to the message queue.

Optionally, after the step of sending the exception message corresponding to the production record and the consumption record to the message queue, the message queue forwards the exception message to the consumption node, the method includes:

the compensation node obtains total retransmission times and single retransmission times of each message;

if the total retransmission times exceeds the first preset times, checking the database utilization rate and the memory utilization rate;

and if the single retransmission times exceeds the second preset times, checking the corresponding message format.

In addition, to achieve the above object, the present invention also provides a message synchronization apparatus including: the system comprises a memory, a processor and a message synchronization program stored on the memory and capable of running on the processor, wherein the message synchronization program is configured to realize the steps of the message synchronization method.

In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a message synchronization program which, when executed by a processor, implements the steps of the message synchronization method.

In one technical scheme provided by the invention, abnormal production records are screened from the production record list, abnormal consumption records are screened from the consumption record list, the production records and the consumption records are sent to a message queue by a compensation node, and then the message queue is forwarded to a consumption node. The scheme provides a unified and complete solution for the problem of inconsistent data of the service system, not only comprises abnormal conditions of a sending stage and a consuming stage, but also the compensation node actually realizes a specific function through a component, so that subsequent operation and maintenance personnel only need to operate and maintain the component, abnormal information can be ensured to be processed rapidly and safely, and the operation and maintenance are convenient and high in efficiency.

Drawings

FIG. 1 is a flow chart of a first embodiment of a message synchronization method according to the present invention;

FIG. 2 is a detailed flowchart of a first embodiment of the message synchronization method of the present invention;

FIG. 3 is a flow chart of a second embodiment of the message synchronization method of the present invention;

FIG. 4 is a flow chart of a third embodiment of the message synchronization method of the present invention;

FIG. 5 is a flow chart of a fourth embodiment of the message synchronization method of the present invention;

FIG. 6 is a flowchart of a fifth embodiment of a message synchronization method according to the present invention;

FIG. 7 is a schematic diagram of components of a message synchronization method of the present invention;

FIG. 8 is a schematic diagram of a compensation module of the message synchronization method of the present invention;

fig. 9 is a schematic structural diagram of a message synchronization device in a hardware running environment according to an embodiment of the present invention.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

Message queuing (hereinafter MQ) is a data structure in the basic data structure that is "first-in first-out," meaning that data (messages) to be transmitted are placed in a queue, and messaging is accomplished by a queuing mechanism-a producer generates a message and places the message in a queue, which is then processed by a consumer. The consumer can pull the message to the appointed queue or subscribe to the corresponding queue, and push the message to the MQ server. The method mainly solves the problems of application decoupling, asynchronous message, flow peak clipping and the like, and realizes a high-performance, high-availability, scalable and final consistency architecture. The main MQ products include: rabbitMQ, activeMQ, rocketMQ, zeroMQ, kafka, IBM WebSphere, etc.

The message queue is used in the service system, so that the effects of asynchronous communication, decoupling system, flow peak clipping, system scalability and reliability improvement and the like can be realized, and the overall performance is further improved.

The MQ provides a transaction mechanism and a message acknowledgement mechanism to support reliable transport of messages, which the business system employs due to poor performance of the transaction mechanism. Although the message confirmation mechanism has high reliability, in the actual use process of the MQ, abnormal situations such as message loss, message repeated consumption and the like are inevitably caused due to various reasons, so that data of a service system are inconsistent. For example, in an intermediate pricing management system, the pricing sheet has been created and in an under-audit state, but the approval task is not seen by the approver.

Several common causes of message loss and repeated consumption are listed here:

(1) As a producer, after a message is sent out, the message is not successfully saved to the MQ and can not be notified to the producer due to internal abnormality of the MQ middleware, and the corresponding message is lost;

(2) As a consumer, the message delivered by the MQ is received and automatically signed, but the consumer does not process business logic, and at the moment, if the consumer service is down, the message is lost;

(3) As a consumer, the message consumption is successful, the business data is submitted, but when the message is signed, the service of the consumer is down or the signing fails due to network reasons, and the MQ can re-deliver the message to the consumer again.

In the related technology, the main solution thinking is that retrying is performed after message confirmation fails, manual intervention is performed after the retrying fails, and meanwhile, consumption-side business logic is required to support idempotent and prevent repeated consumption. However, during this thought practice, there are the following drawbacks:

(1) Different developers aim at the processing thought of the high reliability of the MQ message and the non-unification of specific landing codes, the abnormal scene of the message is often considered to be incomplete, the idempotent processing also respectively depends on the service data for processing, the cost is higher, and a unified and complete solution is lacking;

(2) The code of message processing is scattered in each micro-service of the system, if the code is adjusted, a plurality of micro-services need to be adjusted, and the condition that the code is easy to leak is existed, so that the code maintenance is inconvenient and the efficiency is low;

(3) After the message processing is abnormal, a plurality of platforms are required to analyze reasons and solve problems, the investigation and maintenance are relatively complicated, the efficiency is low, and the monitoring of abnormal messages is lacking.

In order to solve the problems, the application records the sending condition and the consumption condition of the information through the information record table in the database, screens out abnormal information and processes the abnormal information in time, thereby providing a unified and complete solution to ensure the consistency of the data in the system.

In order that the above-described aspects may be better understood, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

The embodiment of the application provides a message synchronization method, which is applied to a business approval system, wherein the business approval system is provided with a message queue, a database, a production node, a consumption node and a compensation node, and referring to fig. 1, fig. 1 is a flow diagram of a first embodiment of the message synchronization method of the application.

In this embodiment, the message synchronization method includes:

Step S11: the compensation node acquires a corresponding production record table and consumption record table from the database;

it is understood that the database is a repository for organizing, storing and managing data according to a data structure, and the scheme can adopt a MySQL database, an Oracle database, an ACCESS database, etc., which is not particularly limited.

In the scheme, the database is in communication connection with the production node, the consumption node and the compensation node, wherein the production node can create a production record table in the database for recording messages sent by the production node to the message queue; the consumption node can create a consumption record table in a database, and the consumption record table is used for recording messages sent by a message queue received by the consumption node; the compensation node may query the production record table and the consumption record table in the database.

Optionally, the various tables in the database are distinguished by table names, each table having a unique name for identifying and accessing the table in the database. The compensation node sends an acquisition instruction to the database through the timing task, or responds to the user operation and maintenance instruction to send the acquisition instruction to the database, the database determines the table names corresponding to the production record table and the consumption record table carried by the acquisition instruction, and then the production record table and the consumption record table are screened out from all tables and returned to the compensation node. The compensation node obtains the production record list and the consumption record list so as to carry out the subsequent abnormality detection and processing.

Optionally, the database sets a timing task to actively send the production record table and the consumption record table to the compensation node.

Step S12: screening production records meeting a first specific state from the production record table, and screening consumption records meeting a second specific state from the consumption record table;

it will be appreciated that the production record table records all message records sent by the production node to the message queue, and this solution defines them as production records for ease of distinction. Because the sending operation of the production node and the receiving operation of the message queue have certain uncertainty, that is, the sending states of the messages are various, and accordingly, each production record is associated with a sending state, for example, sending is in progress, sending is successful, sending is failed, or the retransmission is successful after N times of retransmission and the retransmission is still failed after N times of retransmission, and the embodiment is not limited specifically.

It can be understood that the consumption record table records all message records sent by the message queue received by the consumption node, and for convenience of distinction, the consumption record is defined as the consumption record in the scheme. Similarly, the consumption states of the messages are various, and accordingly, each consumption record is associated with a consumption state, such as a consumption in the middle, a consumption success, and a consumption failure, or a consumption success after N times, and a consumption failure after N times, and the embodiment is not limited specifically.

Optionally, the first feature state is an abnormal state in a transmission stage preset by a technician, for example, the transmission fails, and then the compensation node uses the first feature state as a keyword to perform screening in the production record table, and the finally obtained screening result is a production record meeting the first specific state.

Optionally, the second feature state is an abnormal state of a consumption stage preset by a technician, such as a consumption failure, and then the compensation node uses the second feature state as a keyword to screen in the consumption record table, and the finally obtained screening result is a consumption record meeting the second specific state.

Referring to fig. 2, the step of screening the production record table for the production record satisfying the first specific state includes:

step S121: determining the state types and state holding time length of all production records in the production record table;

step S122: screening the production record with the status type of failed transmission from the production record list; and/or

Step S123: screening out the production record of which the state type is in transmission and the state holding time exceeds the longest waiting time from the production record table;

In this scheme, the first specific state is defined as a transmission failure state, and/or a state in which transmission is in progress for a long period of time.

Optionally, a current status type and a current status holding time period of all production records in the production record table are determined. Screening to obtain abnormal production records by taking the state type as a screening condition of failure in transmission; and/or, taking the state type as the sending state, and the state holding time exceeds the longest waiting time, such as 5 minutes, as the double screening condition, and screening to obtain the abnormal production record.

It should be noted that when the maximum waiting time is set, the use condition of the message queue may be considered. In the case where a message queue clears backlog messages, the waiting time is slightly longer in the normal case, so that the longest waiting time needs to be reasonably determined. Optionally, the compensation node acquires the number of messages in the message queue in real time, and refers to the average processing duration of the historical number of messages, so as to calculate the longest waiting duration.

step S124: determining the state types of all consumption records in the consumption record table;

Step S125: and screening the consumption record with the status type of consumption failure from the consumption record table.

In this scheme, the second specific state is defined as a consumption failure state.

Optionally, the current status type of all consumption records in the message record table is determined. And then screening to obtain an abnormal consumption record by taking the state type as the consumption failure as a screening condition.

Step S13: and sending the abnormal message corresponding to the production record and the consumption record to the message queue, and forwarding the abnormal message to the consumption node by the message queue.

It will be appreciated that it is these abnormal production and consumption records that cause inconsistent data for the business system and therefore require resending or re-consumption.

Optionally, the message corresponding to the production record is originally sent to the message queue by the production node, and in this scheme, in order to improve the exception handling efficiency, the compensation node directly sends the exception message corresponding to the production record to the message queue.

Optionally, the message corresponding to the consumption record is originally sent to the consumer by the message queue, and in the scheme, in order to ensure that the consumption of the message is orderly, the compensation node sends the abnormal message corresponding to the consumption record to the message queue, and the characteristic of first-in first-out of the message queue is utilized, so that the burst flow impact caused by excessive abnormal information is avoided, and the overall stability and expandability are improved.

Further, the message queue forwards the received abnormal information to the consumption node according to the sequence, so that the message synchronization operation is completed, and the data consistency of the system is ensured.

Step A: the compensation node obtains total retransmission times and single retransmission times of each message;

and (B) step (B): if the total retransmission times exceeds the first preset times, checking the database utilization rate and the memory utilization rate;

step C: and if the single retransmission times exceeds the second preset times, checking the corresponding message format.

It will be appreciated that, when the compensating node sends a message each time, the compensating node performs corresponding statistics, and finally obtains the total number of retransmissions. In addition, the compensation node further obtains the primary key information of the transmitted message, and further counts the single retransmission times of each message. Through burying the unusual message of some control, not only can reduce the operation and maintenance personnel's work load to a certain extent, can promote the precision and the efficiency of operation and maintenance moreover, make things convenient for the technical staff to discover the problem in advance and solve.

Alternatively, if the total number of retransmissions exceeds a first predetermined number, e.g., 100, indicating that the micro-service is unstable, then it is necessary to check the database usage and the memory usage, e.g., check whether a bottleneck exists.

Alternatively, if the number of retransmissions exceeds a second predetermined number, such as 10, it is indicated that the message may cause program misplacement, so that attention is paid to the message itself, and the corresponding message format is checked to determine whether the naming convention of the service system is met.

In one technical scheme provided in this embodiment, an abnormal production record is screened from the production record table, an abnormal consumption record is screened from the consumption record table, and then the production record and the consumption record are sent to a message queue by a compensation node, and then forwarded to a consumption node by the message queue. The scheme provides a unified and complete solution for the problem of inconsistent data of the service system, not only comprises abnormal conditions of a sending stage and a consuming stage, but also the compensation node actually realizes a specific function through a component, so that subsequent operation and maintenance personnel only need to operate and maintain the component, abnormal information can be ensured to be processed rapidly and safely, and the operation and maintenance are convenient and high in efficiency.

Further, referring to fig. 3, a second embodiment of the message synchronization method of the present invention is presented. Based on the embodiment shown in fig. 1, before the step of the compensation node obtaining the corresponding production record table and consumption record table from the database, the method includes:

Step S21: after receiving a pricing bill uploaded by a user, the production node sends a corresponding writing instruction to the database, and the database inserts a corresponding production record into a production record table based on the writing instruction, wherein the state type of the production record is in the sending process;

step S22: the production node sends the pricing bill to a message queue and receives a determined signal value returned by the message queue;

it can be understood that the scheme is limited to an intermediate business pricing scenario of a financial institution, a user corresponds to a production node, an approver corresponds to a consumption node, and the whole flow is that the user uploads a pricing sheet to be approved to a business approval system, and then the pricing sheet is sent to the corresponding approver through a message queue.

Optionally, after receiving the pricing sheet uploaded by the user, the production node of the business approval system makes a corresponding record in the database before sending the pricing sheet to the message queue. That is, the production node will send a corresponding write command to the database, and then the database inserts a corresponding production record into the production record table based on the write command, and sets the status type of the production record as being sent.

After this is done, the production node sends the pricing sheet to the message queue.

Further, after receiving the pricing sheet, the message queue generally performs processing such as storing and forwarding, and whether the processing is successful or not, the message queue returns an ACK to the production node, i.e. determines a signal value. Accordingly, the production node receives the determination signal value returned by the message queue.

Step S23: if the confirmation signal value is a first signal value, a first change instruction is sent to the database, and the database changes the state type of the production record into successful sending based on the first change instruction;

step S24: and if the confirmation signal value is a second signal value, sending a second change instruction to the database, and changing the state type of the production record into a sending failure by the database based on the second change instruction.

Optionally, if the signal value is determined to be the first signal value, for example, ack=true, indicating that the message queue is successfully processed, i.e., no exception occurs in the entire transmission phase, at this time, the production node needs to change the corresponding record. That is, the production node sends a first change instruction to the database, and then the database changes the status type of the production record to successful sending based on the first change instruction.

Alternatively, if the signal value is determined to be the second signal value, such as ack=false, the message queue processing fails, i.e. there is an abnormality in the entire transmission phase, at which point the production node needs to change the corresponding record for retransmission. That is, the production node sends a second change instruction to the database, and the database changes the status type of the production record to a failed transmission based on the second change instruction.

In one technical scheme provided in this embodiment, after receiving a pricing sheet uploaded by a user, a production node controls a database to write a new production record into a production record table, then sends the pricing sheet to a message queue, and changes the status type of the production record based on a returned confirmation signal value. The scheme provides writing and changing operation of the production record list, so that complete and accurate basis can be provided for screening abnormal records of subsequent compensation nodes, and the fact that the message is not lost once sent by a message producer is ensured, so that final consistency of service data is ensured.

Further, referring to fig. 4, a third embodiment of the message synchronization method of the present invention is presented. Based on the embodiment shown in fig. 3, before the step of the production node sending the production record to a message queue and receiving the determination signal value returned by the message queue, the method includes:

Step S31: the method comprises the steps that a production node obtains the available capacity of a message queue, and the state type in a production record table is the sending number of consumption records in sending;

step S32: calculating an overflow risk value according to the available capacity and the transmission quantity;

step S33: and if the overflow risk value is lower than a preset risk value, executing the steps that the production node sends the pricing sheet to a message queue and receives a determined signal value returned by the message queue.

It will be appreciated that the capacity of the message queue is limited and there is a risk of overflow, meaning that when the message queue is full, new messages cannot be added to the queue, possibly resulting in a loss of messages or a system crash, and therefore the production node also needs to take into account the use of the message queue before sending the message.

Optionally, the production node checks the status information of the message queue through a management tool or a command line tool, so as to determine the current available capacity, and can store the number of the messages. In addition, the production node acquires a production record table from the database, then screens the production record table by taking the state type as the condition in transmission, and counts the screening result to obtain the corresponding transmission quantity.

Further, according to the available capacity and the number of transmissions, an overflow risk value is calculated, for example, the available capacity is 100, the number of transmissions is 50, and the risk overflow value at this time is 0.5.

Still further, the technician may set a preset risk value, such as 0.8, according to factors such as a specific service system type, a current time, a number of users, and the like. When the overflow risk value is lower than the preset risk value, it indicates that the message queue will not overflow in a short period, so the production node can send more messages to the message queue, that is, execute the steps that the production node sends the pricing sheet to the message queue and receives the determined signal value returned by the message queue.

In one technical solution provided in this embodiment, according to the available capacity of the message queue and the number of transmissions of the consumption record in which the status type is in transmission, an overflow risk value is calculated, and compared with a preset risk value, to determine whether to perform a transmission operation. The method and the device are used for setting the sending rate of the production node according to the available capacity of the current message queue, so that the message list is in a healthy processing rhythm, the condition that the message overflows is avoided, and the effect of continuous approval is achieved.

Further, referring to fig. 5, a fourth embodiment of the message synchronization method of the present invention is presented. Based on the embodiment shown in fig. 3, after receiving the pricing bill uploaded by the user, the production node sends a corresponding writing instruction to the database, and the database inserts a corresponding production record into the production record table based on the writing instruction, where the status type of the production record is that after the step in sending, the method includes:

step S41: determining a production record element of the production record;

step S42: acquiring consumption record elements of all consumption records in the consumption record table;

step S43: and if the consumption pricing element which is the same as the production pricing element exists, sending a third changing instruction to the database, and changing the state type of the production record into a state type which does not need to be sent based on the third changing instruction by the database.

It will be appreciated that each production record is made up of a number of production record elements, such as pricing sheets, including but not limited to tolls, billing elements, customer information, product names, raw material costs, transportation costs, sales areas, etc.

Optionally, after inserting the corresponding production record into the production record table, the production node control database determines the production record elements of the production record, and then obtains the consumption record elements of all the consumption records in the consumption record table.

Further, comparing the production record element with all the consumption record elements, if there is an identical consumption record element, i.e. there is an identical consumption record to the production record, this situation indicates that the pricing sheet currently submitted by the user has been submitted before, without requiring a secondary approval. Therefore, the production node sends a third change instruction to the database, and the database changes the status type of the production record to no need of sending based on the third change instruction.

In addition, the production node can also send the reminding information of repeated approval to the user, and feed back the approval result of the last time to the user.

In one technical solution provided in this embodiment, the production record elements of the production record are compared with the consumption record elements of all the consumption records in the consumption record table, and the status of the production record is updated according to the comparison result. By the arrangement, the pricing orders repeatedly submitted by the users can be effectively identified, the approval process of the pricing orders is saved, the processing amount of the message queue is further reduced, and the approval rate of other effective pricing orders is accelerated.

Further, referring to fig. 6, a fifth embodiment of the message synchronization method of the present invention is presented. Based on the embodiment shown in fig. 1, before the step of the compensation node obtaining the corresponding production record table and consumption record table from the database, the method includes:

Step S51: after receiving the pricing bill sent by the message queue, the consumption node sends a corresponding writing instruction to a database, and the database inserts a corresponding consumption record into a consumption record table based on the writing instruction;

step S52: the database judges whether the primary key of the consumption record has conflict or not;

step S53: if the conflict exists, setting the state type of the consumption record as repeated consumption;

step S54: and if no conflict exists, setting the state type of the consumption record as consumption.

It will be appreciated that the present solution is limited to an intermediate business pricing scenario for a financial institution, in which a message queue would send pricing sheets to consuming nodes in chronological order.

Optionally, after receiving the pricing bill sent by the message queue, the consuming node may perform corresponding records in the database. That is, the consuming node sends a corresponding writing instruction to the database, and then the database inserts a corresponding consuming record into the consuming record table based on the writing instruction.

Further, the database may determine whether the above insertion behavior is valid, specifically, based on the primary key uniqueness constraint of each consumption record, determine whether the primary key of the consumption record is the same as the primary key of the previous consumption record, that is, the primary key conflicts, and if the business logic is continuously executed, trigger idempotent to cause repeated consumption.

Based on the principle, if a conflict exists, the state type of the consumption record is set to be repeated consumption, the subsequent consumption node directly confirms the information and does not execute business logic; if no conflict exists, the state type of the consumption record is set as the consumption, and the subsequent consumption nodes execute service logic on the information.

Step S55: the consumption node screens out the consumption record of which the state type is in consumption, and executes corresponding business logic on a pricing bill corresponding to the consumption record;

step S56: if the service logic is executed successfully, a fourth change instruction is sent to the database, and the database changes the state type of the consumption record into consumption success based on the fourth change instruction and returns a corresponding confirmation signal value to the message queue;

step S57: and if the execution service logic fails, a fifth change instruction is sent to the database, and the database changes the state type of the consumption record into consumption failure based on the fifth change instruction and returns a corresponding confirmation signal value to the message queue.

Optionally, after the database completes the status update, the consumption node screens out the consumption record with the status type being in consumption, and executes corresponding business logic on the corresponding pricing sheet, such as creating a pricing sheet approval instance, and initializing pricing sheet data for approval by an approver.

Further, if the execution of the business logic is successful, it indicates that the processing of the consuming node is successful, i.e. no exception exists in the whole consuming stage, and at this time, the consuming node needs to change the corresponding record. That is, the consuming node sends a fourth change instruction to the database, and then the database changes the status type of the consumption record to be successful based on the fourth change instruction, and returns a corresponding acknowledgement signal value, that is, ack=true, to the message queue.

Further, if the execution of the business logic fails, it indicates that the processing of the consuming node fails, i.e. there is an exception in the whole consuming stage, at this time, the consuming node needs to change the corresponding record so as to consume again. That is, the consuming node sends a fifth change instruction to the database, and then the database changes the status type of the consumption record to a consumption failure based on the fifth change instruction, and returns a corresponding acknowledgement signal value, that is, ack=false, to the message queue.

In one technical scheme provided in this embodiment, after the database inserts the corresponding consumption record into the consumption record table, the state type is set for the consumption record table according to the primary key conflict condition of the message record, and in addition, the state type is changed according to the execution result of executing the service logic. The scheme provides writing and changing operation of the production record list, so that complete and accurate basis can be provided for screening abnormal records of subsequent compensation nodes, the fact that after the information is received as a consumer is ensured, the information is not lost, the information is not consumed repeatedly, and the final consistency of service data is ensured.

From the technical and resource intensive point of view, the modules corresponding to each node are combined into one overall assembly, as shown in fig. 7.

The compensation module can be subdivided into an automatic compensation module and a manual operation and maintenance module, referring to fig. 8, the automatic compensation means that the compensation node automatically processes abnormal information in the production record table and the consumption record table through a timing task; the manual operation and maintenance means that operation and maintenance personnel manually perform operations such as adding, editing, deleting, retransmitting and the like on the message through a front-end operation and maintenance interface, and it is noted that a front-end page is automatically assembled into a compensator module by using a boot-starter so as to reduce unnecessary configuration of a user.

Referring to fig. 8, the compensation module is connected to a database of several business micro services, such as login micro services, approval micro services, etc., and specifically adopts a configuration mode to inform the compensation module of which micro services need to be compensated for retransmission. As for the interaction mode of the compensation module and the business micro-service, a database direct connection mode is specifically adopted to obtain a record table of the micro-service.

In addition, the logic of message query, CRUD, resend, etc. is the same for each microservice, with the only difference being the transaction manager and the data source. We can dynamically inject the compensated services into the Spring container by servicescan configuration < service name, service transaction manager >, and implement message query, CRUD, and resend logic for different data sources, respectively, by generic class MessageManualOpsService, messageAutoOpsService.

The direct connection mode has the following characteristics: after the compensator is configured with the service name and the service transaction manager is mapped, the compensation function is directly realized through the operation database; the performance is good, and the service micro-service resource is not occupied.

It should be noted that the core classes of the present solution include Secure Producer, secure Consumer, and MQ Messages Resend Task, an auto compensator class, MQ Manager API.

The Secure Producer is provided for a reliable sender of client messages used by producers, and the messages are sent by the Secure setting method, so that the messages of the production end can be ensured not to be lost after being sent to the MQ server;

the Secure Consumer is provided for a reliable client message receiver used by a Consumer, and the Secure Consumer receives and processes the message by the Secure Consumer method, so that the message received by the Consumer can be ensured not to be lost, and meanwhile, the message is subjected to idempotent processing;

MQ Messages Resend Task the automatic compensator core class, the failure message resends the timing task, configures the automatic triggering time through the configuration item bee. Schedule-end-job. Cron, and the automatic task resends the data base failure message of the compensated service through reading;

The MQ Manager API, the manual compensator core class, provides an API interface and a WEB operation page for operation and maintenance personnel to operate in fine granularity. The API comprises interfaces of a compensated service list query interface, a message sending record list interface, a modified sending record, a deleted sending record, a production message retransmission, a message consumption list interface, a modified consumption record, a deleted consumption record, a message re-consumption, a custom sending new message and the like.

It should be noted that, through the embedded monitoring points of two indexes for monitoring and alarming abnormal information, the platform configuration monitoring rule in the management information center realizes the decoupling of the event and the alarm. The embedded point code is in the production module and provides an expansion interface IMqRepeatNum, IRetryCount, so that the user can customize the alarm action.

The imqrepeat embedded point index is used for monitoring total retransmission times, and the IRetryCount embedded point index is used for monitoring single retransmission times.

It should be noted that the present invention is also striving for ease of use while focusing on the MQ reliability and data consistency aspects of the complete solution. The integration of the service system only requires the following steps:

first, configure component dependencies

Gradle relies on: completion ('LZ 62: rubbitmq-secure: 1.0.0')

Second, initialize the database table structure, create the local message table of the producer or consumer, mysql script in the rubbifre module sql directory mysql_init_sql file

Third, enabling producer and consumer to start component functions through @ EnableSecureMQ annotation

Fourth step, send or receive message

Fifth, the compensator is enabled and the compensated service is registered, and the compensator is started through @ EnableSecureMQOps

Sixth, the compensator micro-service needs to directly connect to the database of the compensated and uses ZA21-Scheduler for timing tasks, so the compensator micro-service needs to configure multiple data sources and ZA21-Scheduler related configuration.

ZA21-Scheduler related configurations are described in ZA21 development manual. Two timing tasks are realized in the component, the triggering time of the two timing tasks needs to be configured, and the configuration items are as follows:

the failure message resends the task bee. Scheduler. Rabit-end-job cron

The success message deletes the task bee.scheduler. Rabit-success-delete-job cron

Seventh, the front end is accessed

Independent identity authentication: using a username and password login independent of the business system, the username and password can be configured in a configuration with default front-end addresses of: compensator micro-service host+ "/mq-management/index. Html'

Business system token identity authentication: using token identity authentication of the service system, default address: host+ "/mq-management/index. Htmltype=header & key=xxx & value=xxx", where key is the token name of the business system and value is the token value of the business system

Referring to fig. 9, fig. 9 is a schematic diagram of a message synchronization device structure of a hardware running environment according to an embodiment of the present invention.

As shown in fig. 9, the message synchronization device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a WIreless interface (e.g., a WIreless-FIdelity (WI-FI) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

It will be appreciated by those skilled in the art that the structure shown in fig. 9 does not constitute a limitation of the message synchronization device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

As shown in fig. 9, an operating system, a data storage module, a network communication module, a user interface module, and a message synchronization program may be included in the memory 1005 as one type of storage medium.

In the message synchronization device shown in fig. 9, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the message synchronization device of the present invention may be disposed in the message synchronization device, and the message synchronization device invokes a message synchronization program stored in the memory 1005 through the processor 1001 and executes a message synchronization method provided by an embodiment of the present invention.

Embodiments of the present invention provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of any of the embodiments of the message synchronization method described above.

Since the embodiments of the computer readable storage medium portion and the embodiments of the method portion correspond to each other, the embodiments of the computer readable storage medium portion are referred to the description of the embodiments of the method portion, and are not repeated herein.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims

1. The message synchronization method is characterized by being applied to a business approval system, wherein the business approval system is provided with a message queue, a database, a production node, a consumption node and a compensation node, and comprises the following steps:

2. The message synchronization method of claim 1, wherein the step of screening production records from the production record table that satisfy a first particular state comprises:

3. The message synchronization method of claim 1, wherein prior to the step of the compensation node retrieving the corresponding production record table and consumption record table from the database, comprising:

4. A method of synchronizing messages as claimed in claim 3, wherein said production node, prior to the step of sending said production record to a message queue and receiving a determined signal value returned by said message queue, comprises:

5. The message synchronization method as claimed in claim 3, wherein after receiving the pricing sheet uploaded by the user, the production node sends a corresponding writing instruction to the database, and the database inserts a corresponding production record into the production record table based on the writing instruction, and the status type of the production record is that after the step of sending, the method comprises:

Determining a production record element of the production record;

6. The message synchronization method according to any of claims 1-5, wherein before the step of the compensation node obtaining the corresponding production record table and consumption record table from the database, it comprises:

7. The message synchronization method according to claim 6, wherein after the step of setting the status type of the consumption record to be in consumption if there is no conflict, comprising:

8. The message synchronization method according to claim 1, wherein the step of sending the exception message corresponding to the production record and the consumption record to the message queue, the message queue forwarding the exception message to the consumption node, comprises:

9. A message synchronization device, the message synchronization device comprising: memory, a processor and a message synchronization program stored on the memory and executable on the processor, the message synchronization program being configured to implement the steps of the message synchronization method according to any one of claims 1 to 8.

10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a message synchronization program, which when executed by a processor, implements the steps of the message synchronization method according to any of claims 1 to 8.