CN114969201A

CN114969201A - Data synchronization method and device, storage medium and electronic equipment

Info

Publication number: CN114969201A
Application number: CN202210411735.8A
Authority: CN
Inventors: 张娜; 于灏; 刘震; 杨猛; 潘曦; 马南; 周书橙
Original assignee: Peoples Insurance Company of China
Current assignee: Peoples Insurance Company of China
Priority date: 2022-04-19
Filing date: 2022-04-19
Publication date: 2022-08-30

Abstract

The application discloses a data synchronization method and device, a storage medium and an electronic device, wherein the method comprises the following steps: acquiring changed data in a first database, wherein the changed data comprises pre-changed data and post-changed data; determining a change value according to the data before change and the data after change; and updating the second database according to the change value. The method is suitable for the scene that final data needs to be obtained through calculation according to the basic data and the changed process data, and is simple in calculation and high in efficiency.

Description

Data synchronization method and device, storage medium and electronic equipment

Technical Field

The present application relates to the field of data synchronization technologies, and in particular, to a data synchronization method, an apparatus, a storage medium, and an electronic device.

Background

The data of the service systems are stored in respective databases, and the systems and the data are vertical and independent. In the background of the digital era, all services have the online characteristic, a database stores a large amount of data with various dimensions such as clients, policy, services and the like, and the data can respectively support own systems such as client systems, underwriting systems and the like, but now along with the technological progress and development, if the data can be fused, the data can be mined to have larger and more important value, so that each system wants to break the data 'island', the data summarization and fusion are realized, and a data synchronization technology is generated, and is used for synchronizing the data in one database to another data storage medium with different possible types accurately in real time.

In the conventional data synchronization method, because the database only records currently changed data, new data after being changed is only synchronized to the computing platform during data synchronization, and the computing platform cannot know the data before being changed, so that the data change condition cannot be known.

Disclosure of Invention

In view of this, the present application provides a data synchronization method, apparatus, medium, and device, which are suitable for a scene in which final data needs to be obtained by calculation according to basic data and changed process data, and are simple in calculation and high in efficiency.

According to an aspect of the present application, there is provided a data synchronization method, including:

acquiring changed data in a first database, wherein the changed data comprises pre-changed data and post-changed data;

determining a change value according to the data before change and the data after change;

and updating the second database according to the change value.

Optionally, before determining a change value according to the pre-change data and the post-change data, the method further includes:

splicing the change data to obtain a change message, and adding the change message into a message queue;

and if the message queue has the change message, reading and analyzing the change message in the message queue to obtain the change data.

Optionally, the splicing the change data to obtain a change packet specifically includes:

determining a change type according to the change data, and writing the change type into a type field, wherein the change type is one of the following types: adding, deleting and modifying;

writing the data before the change into a field before the change and writing the data after the change into a field after the change;

and splicing the type field, the field before the change and the field after the change to obtain the change message.

Optionally, the obtaining of the changed data in the first database specifically includes:

monitoring a data log of the first database;

and if the data change record written in the data log is monitored, extracting the change data corresponding to the data change record in the data log.

Optionally, the data log is configured to record the data change record and a data history version;

the extracting of the change data corresponding to the data change record in the data log specifically includes:

determining a data history version corresponding to the data change record, wherein the data history version comprises a version before change and a version after change;

and determining the changed data according to the historical data version.

reading a unique identifier carried by the changed data, and judging whether the changed data is processed or not according to the unique identifier and a history processing record;

if not, updating the history processing record according to the unique identifier;

if yes, generating abnormal information and ending.

Optionally, the reading and analyzing the change packet in the message queue specifically includes:

and if a plurality of change messages exist in the message queue, reading the change messages according to a first-in first-out sequence, and analyzing the read change messages.

According to another aspect of the present application, there is provided a data synchronization apparatus including:

the acquisition module is used for acquiring changed data in a first database, wherein the changed data comprises data before change and data after change;

the calculation module is used for determining a change value according to the data before change and the data after change;

and the updating module is used for updating the second database according to the change value.

Optionally, the apparatus further includes a packet processing module, specifically configured to:

Optionally, the message processing module is further configured to:

Optionally, the obtaining module is further configured to:

monitoring a data log of the first database;

the obtaining module is further configured to:

and determining the changed data according to the historical data version.

Optionally, the apparatus further comprises a verification module, specifically configured to:

if yes, generating abnormal information and ending.

Optionally, the message processing module is further configured to:

According to yet another aspect of the present application, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described data synchronization method.

According to yet another aspect of the present application, there is provided an electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, the processor implementing the above data synchronization method when executing the computer program.

By means of the technical scheme, a synchronization method of only synchronizing changed data is abandoned, the data before the change and the data after the change are obtained at the same time, the change value is determined according to the data before the change and the data after the change, the updating operation can be achieved according to the change value, the first database does not need to be inquired for historical data every time, the database operation process is simplified, and the efficiency is improved.

The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a schematic flowchart illustrating a data synchronization method according to an embodiment of the present application;

fig. 2 is a schematic flow chart illustrating another data synchronization method provided in an embodiment of the present application;

fig. 3 is a schematic flowchart illustrating another data synchronization method provided in an embodiment of the present application;

FIG. 4 is a flow chart illustrating another data synchronization method provided by an embodiment of the present application;

fig. 5 is a schematic flowchart illustrating another data synchronization method provided in an embodiment of the present application;

fig. 6 is a schematic flow chart illustrating another data synchronization method provided in the embodiment of the present application;

fig. 7 shows a data flow diagram of a data synchronization method provided by an embodiment of the present application;

FIG. 8 is a data flow diagram illustrating another data synchronization method provided by an embodiment of the present application;

fig. 9 shows a block diagram of a data synchronization apparatus according to an embodiment of the present application.

Detailed Description

In this embodiment, a data synchronization method is provided, as shown in fig. 1, the method includes:

step 101, obtaining changed data in a first database, wherein the changed data comprises data before change and data after change;

the data synchronization method provided by the embodiment of the application is suitable for a scene needing to obtain final data through calculation according to basic data and changed process data. By the data before and after being changed, the data change situation can be known, and the historical data can be known or the change situation of the data can be known without inquiring the historical data. Based on the above, the data before the change and the data after the change are firstly obtained in the first database, so as to use the data to know the data change situation, wherein the first database is the database of the upstream system.

102, determining a change value according to the data before change and the data after change;

and 103, updating the second database according to the changed value.

In this embodiment, the data before the change and the data after the change are comprehensively analyzed, so that the data change can be known, that is, the change value is determined, and then the second database is updated according to the change value. Wherein, the second database is the database of the downstream system, the changed data is synchronous between the first database and the second database, namely, the data transmission of the upstream system and the downstream system is realized,

by applying the technical scheme of the embodiment, a synchronization method of only synchronizing changed data is abandoned, the data before the change and the data after the change are obtained at the same time, the change value is determined according to the data before the change and the data after the change, the update operation can be realized according to the change value without inquiring historical data from the first database every time, the database operation process is simplified, and the efficiency is improved.

Further, as a refinement and an extension of the specific implementation of the above embodiment, in order to fully describe the specific implementation process of the embodiment, another data synchronization method is provided, as shown in fig. 2, before determining a change value according to pre-change data and post-change data, the method further includes:

step 201, splicing the change data to obtain a change message, and adding the change message into a message queue;

step 202, if it is identified that there is a change message in the message queue, reading and analyzing the change message in the message queue to obtain change data.

In this embodiment, the data before the change, the data after the change, and other information related to the change are concatenated to obtain a change message. The messages are added into the message queue, and then the messages are read from the message queue, and the message queue is utilized to realize buffering, so that the problems of system crash and the like caused by inconsistent efficiency of data processing of upstream and downstream systems are avoided.

Specifically, the message queue used in the present application is Kafka, which is a distributed, high-throughput, highly scalable message queue system. The message queue is a storage container, the upstream system serves as a producer to send messages (namely change messages) to the storage container and store the messages, and the downstream system serves as a consumer to acquire the messages from the storage container and perform the next processing according to the requirements or the processing capacity of the downstream system.

The scheme uses the Kafka message queue to realize the decoupling of an upstream system and a downstream system, so that the upstream system and the downstream system cannot be influenced by different processing capacities. Specifically, when the upstream traffic is large, the data can be stored in the message queue and the self-service processing can be rapidly carried out, so that the data cannot be dragged down by a downstream system; when the processing capacity of the downstream system is limited, the data can be slowly acquired from the message queue, and the data cannot be crushed by the upstream system.

Further, as shown in fig. 3, in another data synchronization method, splicing the changed data to obtain a changed packet specifically includes:

step 301, determining a change type according to the change data, and writing the change type into a type field, wherein the change type is one of the following types: adding, deleting and modifying;

in this embodiment, for the data of the database, the conventional operations include addition, deletion, modification and check, where addition represents inserting data into the database table, deletion represents deleting data in the database table, modification represents changing the value of the data in the database table, and table lookup represents querying the data in the database table. The data can be changed by adding, deleting and changing, so that the corresponding change type can be determined according to the changed data.

Specifically, for example, if the change type is incremental, insert may be written to the type field; if the change type is delete, delete can be written into the type field; if the change type is changed, update may be written to the type field. Of course, other flags may be set as the flag of the change type in addition to the insert, delete, and update, which is not limited herein.

Step 302, writing the data before the change into the field before the change, and writing the data after the change into the field after the change;

in this embodiment, the pre-change data and the post-change data are written into the pre-change field and the post-change field, respectively, in accordance with the method for writing the change type into the type field described above.

And 303, splicing the type field, the field before the change and the field after the change to obtain the change message.

In this embodiment, a plurality of fields written with specific data values are concatenated to obtain a change message. In addition to the type field, the field before the change, and the field after the change, the change time field, the single number field, the unique identifier field, and the like may also be spliced, which is not limited herein.

For example, if only changed data is synchronized, the synchronized message content is as follows:

by adopting the synchronization method, a plurality of service scenes cannot be met. For example, if the premium amount is accumulated and calculated, the premium amount of a certain policy is 1000 yuan at the time of the first transaction, and the policy is modified for some reason, the premium amount of the policy (i.e., the amount in the above-mentioned synchronization message) becomes 1800 yuan, and then if the downstream computing system adds the second message and the first message to obtain that the premium of the policy is 2800 yuan, it is obviously incorrect, and the downstream computing system needs to know the variation amount of the two messages, i.e., 800 yuan, and then can accumulate and calculate the correct premium amount based on 1000 yuan of the first message.

If the method of this embodiment is adopted to splice the changed type, the data before the change, the data after the change, and the like and then synchronize, the synchronized message content may be as follows:

in the synchronous message, the type of change is update, namely the type of change; before data is in before { }, and after data is in after-alteration. For example, the name field is modified from wang before the change to zhang after the change.

Further, in another data synchronization method, as shown in fig. 4, acquiring change data in the first database specifically includes:

step 401, monitoring a data log of a first database;

step 402, if it is monitored that a data change record is written in the data log, extracting changed data corresponding to the data change record from the data log.

In this embodiment, the changed data is obtained in real time by listening to the data log of the first database. Specifically, operations such as addition, deletion, modification and check on the first database write corresponding records into the data log, and if it is monitored that a data change record is written into the data log, it can be considered that a change operation is performed on the data in the first database, and based on this, changed data corresponding to the data change record is extracted from the data log.

Furthermore, after the changed data is extracted, the changed data can be converted into a json format and then added into a message queue, and the json occupies smaller storage space and is easier to analyze, so that the space occupation can be reduced and the data synchronization efficiency can be improved.

The embodiment abandons the existing method of operating Data by using jdbc (Java Data Base Connectivity, Java database connection) to drive and connect the database, and acquires Data change in real time by monitoring the underlying Data log, so that the method does not contend for resources with normal service connection, and the influence on the performance of the database is very small.

Further, in another data synchronization method, the data log is configured to record a data change record and a data history version;

as shown in fig. 5, extracting the change data corresponding to the data change record from the data log specifically includes:

step 501, determining a data history version corresponding to a data change record, wherein the data history version comprises a version before change and a version after change;

step 502, determining the changed data according to the historical version of the data.

In the embodiment, the data log comprises a historical version of data generated by each change, if a data change record is monitored to be written in the data log, a pre-change version and a post-change version corresponding to the current change are determined according to the data change record, before-change data is extracted from the pre-change version, and after-change data is extracted from the post-change version.

Further, in another data synchronization method, as shown, before determining a changed value according to the pre-changed data and the post-changed data, the method further includes:

601, reading a unique identifier carried by the changed data, and judging whether the changed data is processed according to the unique identifier and a history processing record;

step 602, if not, updating the history processing record according to the unique identifier;

and step 603, if so, generating abnormal information and ending.

In this embodiment, the change data may carry a unique identifier, and this unique identifier may indicate a change operation corresponding to the change data. Before processing the changed data to obtain the changed value, a record related to the unique identifier can be searched in a history processing record, and whether the changed data is processed or not can be judged by using the unique identifier. If the data is processed, the data is not processed again, and the synchronous operation aiming at the changed data is directly finished; if not, the change data continues to be processed and a record associated with the unique identifier is inserted into the history record.

According to the embodiment, each piece of changed data is processed only once according to the unique identification, and the condition of processing omission or repeated processing is avoided.

Further, in another data synchronization method, reading and analyzing a change packet in a message queue specifically includes:

and if a plurality of change messages exist in the message queue, reading the change messages according to the first-in first-out sequence, and analyzing the read change messages.

In this embodiment, if the speed of the upstream system generating the change data is faster than the speed of the downstream system synchronously changing the data, the change message accumulation condition will occur in the message queue. At this time, the change messages in the message queue can be sequentially read and processed according to the first-in first-out order, so as to avoid the disorder of data in the processing process.

The embodiment realizes the streaming processing of the changed data by using the message queue, and is suitable for the borderless data sets such as the logs of the server, the sensor signal data and the like. Here, the borderless data set means that new data is continuously generated from the beginning of data generation, and thus the data has no boundary.

Further, fig. 6 shows the overall architecture of another data synchronization method. As shown in the figure, firstly, the foundation for supporting the whole process flow processing is provided, in this step, the data log of the first database is monitored to capture the changed data in real time, rather than being connected through jdbc, so that the influence on the performance of the underwriting database (i.e. the first database) and the underwriting system (i.e. the upstream system) is very small; the content of the changed data comprises data before the change and data after the change, and the changed data is pushed to the message queue. And the second is a message queue which is used for realizing the decoupling of an upstream system and a downstream system and can ensure the sequence of the changed data. Thirdly, the stream processing program has the following characteristics: calculating to obtain changed contents through the data before change and the data after change in the changed message; the operation is only performed once when the statistical platform database (namely the second database) is connected, and the performance is very good; the method has fault tolerance, can flexibly process various errors and ensures high availability of the process; the method has idempotency, and ensures that each message is processed only once according to unique identification such as a single number, a serial number and the like.

Further, FIG. 7 illustrates a data flow diagram of a data synchronization method that does not contain pre-change data in one particular scenario; fig. 8 shows a data flow diagram of a data synchronization method including pre-change data provided by the present application in this scenario.

Specifically, as shown in fig. 7, in the underwriting process, the business system records the premium a0 of the policy a in the underwriting database (i.e. the first database) of the underwriting system (i.e. the upstream system), and the data is synchronized to the statistical platform in real time, so that the statistical platform counts the premium income of the entire company in real time. The whole process is as follows:

(1) the client purchases insurance, generates insurance policy A and pays, and the premium amount of the transaction is a 0;

(2) in a certain consideration, the client modifies the original insurance policy A and adds some insurance items, so that the premium is increased, the new premium amount is a1, and the spliced change message only contains a 1;

(3) in some consideration, the customer makes a correction again, reduces some insurance items, the premium amount becomes a2, and the spliced change message only contains a 2.

In the data synchronization method without data before change, when the statistical platform (i.e. downstream system) receives the data of step (2), it needs to query whether there is data a0 in the statistical platform database (i.e. second database), if so, it is deleted, and then the data a1 is used as the standard; when the data of step (3) is received, it is required to inquire whether there is data a1 or not, and if so, delete it, and then take a2 as the standard. The steps are complex and the performance is low. .

As shown in fig. 8, in the data synchronization method including the data before change, the entire data synchronization contents are as follows:

(2) in some consideration, the customer modifies the original policy A and adds some insurance items, so that the premium is increased, the new premium amount is a1, the data before the change is a0, and the data after the change is a1, so that the spliced change message contains both a0 and a 1;

(3) in some consideration, the client makes a correction again, reduces some insurance items, the premium amount becomes a2, the data before the change is a1, and the data after the change is a2, so the spliced change message contains both a1 and a 2.

After the statistical platform (i.e. the downstream system) acquires the change data each time, it can know whether the record has the historical version, and what the historical version and the current version are changed according to the change data. For example, when the data of step (2) is received, it is determined that there is a history version based on the data a0 before the change, and the amount of change between the history version and the current version, that is, the change value is determined based on the data a0 before the change and the data a1 after the change. Thus, only one operation of connecting the second database is needed each time. Simple steps and high performance.

Further, as a specific implementation of the data synchronization method, an embodiment of the present application provides a data synchronization apparatus, as shown in fig. 9, where the data synchronization apparatus includes: the device comprises an acquisition module, a calculation module and an updating module.

The acquisition module is used for acquiring changed data in the first database, wherein the changed data comprises data before change and data after change;

the calculation module is used for determining a change value according to the data before the change and the data after the change;

In a specific application scenario, optionally, the apparatus further includes a message processing module, specifically configured to:

splicing the changed data to obtain a changed message, and adding the changed message into a message queue;

and if the message queue is identified to have the change message, reading and analyzing the change message in the message queue to obtain the change data.

In a specific application scenario, optionally, the message processing module is further configured to:

writing the data before the change into the field before the change and writing the data after the change into the field after the change;

and splicing the type field, the field before the change and the field after the change to obtain a change message.

In a specific application scenario, optionally, the obtaining module is further configured to:

monitoring a data log of a first database;

and if the data change record is monitored to be written into the data log, extracting the change data corresponding to the data change record from the data log.

In a specific application scenario, optionally, the data log is configured to record a data change record and a data history version;

an acquisition module further configured to:

and determining changed data according to the historical data version.

In a specific application scenario, optionally, the apparatus further includes a verification module, specifically configured to:

reading the unique identifier carried by the changed data, and judging whether the changed data is processed according to the unique identifier and the history processing record;

if not, updating the historical processing record according to the unique identifier;

if yes, generating abnormal information and ending.

According to yet another aspect of the present application, there is provided a storage medium having stored thereon a computer program, the program or instructions when executed by a processor implementing the above-described data synchronization method.

According to yet another aspect of the present application, there is provided an electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the data synchronization method when executing the computer program.

It should be noted that other corresponding descriptions of the functional modules related to the data synchronization apparatus provided in the embodiment of the present application may refer to the corresponding descriptions in fig. 1 to fig. 8, and are not described herein again.

Based on the methods shown in fig. 1 to 8, correspondingly, the present application further provides a storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data synchronization method shown in fig. 1 to 8.

Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.), and includes several instructions for enabling an electronic device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the implementation scenarios of the present application.

Based on the foregoing methods shown in fig. 1 to fig. 8 and the data synchronization apparatus embodiment shown in fig. 9, in order to achieve the foregoing object, an embodiment of the present application further provides an electronic device, which may specifically be a personal computer, a server, a network device, and the like, where the electronic device includes a storage medium and a processor; a storage medium for storing a computer program; a processor for executing a computer program to implement the data synchronization method as described above with reference to fig. 1 to 8.

Optionally, the electronic device may further include a user interface, a network interface, a camera, Radio Frequency (RF) circuitry, sensors, audio circuitry, a WI-FI module, and so forth. The user interface may include a Display screen (Display), an input unit such as a keypad (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (e.g., a bluetooth interface, WI-FI interface), etc.

It will be understood by those skilled in the art that the present embodiment provides an electronic device structure that is not limiting of the electronic device, and may include more or fewer components, or some components in combination, or a different arrangement of components.

The storage medium may further include an operating device and a network communication module. The operation device is a program for managing and saving hardware and software resources of the electronic device, and supports the operation of the information processing program and other software and/or programs. The network communication module is used for realizing communication among the controls in the storage medium and communication with other hardware and software in the entity equipment.

Through the above description of the embodiments, those skilled in the art will clearly understand that the present application can be implemented by software plus a necessary general hardware platform, and can also be implemented by hardware.

Those skilled in the art will appreciate that the drawings are merely schematic representations of one preferred implementation scenario and that the elements or processes in the drawings are not necessarily required to practice the present application. Those skilled in the art will appreciate that elements of a device in an implementation scenario may be distributed in the device in the implementation scenario according to the description of the implementation scenario, or may be located in one or more devices different from the present implementation scenario with corresponding changes. The units of the implementation scenario may be combined into one unit, or may be further split into a plurality of sub-units.

The above application serial numbers are for description purposes only and do not represent the superiority or inferiority of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present application, but the present application is not limited thereto, and any variations that can be made by those skilled in the art are intended to fall within the scope of the present application.

Claims

1. A method for synchronizing data, the method comprising:

and updating the second database according to the change value.

2. The method of claim 1, wherein prior to determining a change value based on the pre-change data and the post-change data, further comprising:

3. The method according to claim 2, wherein the splicing the change data to obtain a change packet specifically comprises:

4. The method according to claim 1, wherein the obtaining of the change data in the first database specifically comprises:

monitoring a data log of the first database;

5. The method of claim 4, wherein the data log is configured to record the data change record and a data history version;

and determining the changed data according to the historical data version.

6. The method of claim 1, wherein before determining a change value based on the pre-change data and the post-change data, the method further comprises:

if yes, generating abnormal information and ending.

7. The method according to claim 2, wherein the reading and parsing the change packet in the message queue specifically includes:

8. A data synchronization apparatus, the apparatus comprising:

9. A storage medium having a program or instructions stored thereon, which when executed by a processor, performs the steps of the method of any one of claims 1 to 7.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when executing the program.