CN117435668A

CN117435668A - Data synchronization method, device, computer equipment and storage medium

Info

Publication number: CN117435668A
Application number: CN202311227042.4A
Authority: CN
Inventors: 柴雨桐; 赵润泽; 任东胜
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2023-09-21
Filing date: 2023-09-21
Publication date: 2024-01-23

Abstract

The present application relates to a data synchronization method, apparatus, computer device, storage medium and computer program product. The method comprises the following steps: acquiring a data table operation notification sent by a message queue subscribed by a second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; acquiring a synchronous data table field set which is preconfigured for a second data table in a first data table; under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table; the second data table is updated based on the second data table operation. The method belongs to the technical field of big data, and can improve the efficiency of data synchronization.

Description

Data synchronization method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technology, and in particular, to a data synchronization method, apparatus, computer device, storage medium, and computer program product.

Background

With the development of computer technology, a large number of databases are available for selection, each database has advantages and disadvantages, and a proper type of database can be selected by combining the service scene, the data volume and the like of the system. In this context, when performing data processing for online services, it is often necessary to implement data synchronization between data tables by using a data synchronization technique.

The traditional data synchronization method is to read database logs at regular time, identify changed data in a data table based on the database logs, and synchronize the changed data to a related data table, so that the problem of low data synchronization efficiency exists.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data synchronization method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve data synchronization efficiency.

The application provides a data synchronization method. The method comprises the following steps:

acquiring a data table operation notification sent by a message queue subscribed by a second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases;

Acquiring a synchronous data table field set which is preconfigured for a second data table in a first data table;

under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table;

based on the second data table operation, the second data table is updated.

The application also provides a data synchronization device. The device comprises:

the operation notification acquisition module is used for acquiring a data table operation notification sent by a message queue subscribed by the second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases;

the synchronous field set acquisition module is used for acquiring a synchronous data table field set preset for the second data table in the first data table;

the data table operation conversion module is used for converting the first data table operation based on a database operation conversion rule aiming at the first data table operation under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, so as to obtain a second data table operation aiming at the second data table;

And the data table updating module is used for updating the second data table based on the second data table operation.

A computer device comprising a memory storing a computer program and a processor implementing the steps of the data synchronization method described above when the processor executes the computer program.

A computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the data synchronization method described above.

A computer program product comprising a computer program which, when executed by a processor, performs the steps of the data synchronization method described above.

The data synchronization method, the data synchronization device, the computer equipment, the storage medium and the computer program product are used for acquiring the data table operation notification sent by the message queue subscribed by the second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases. And acquiring a synchronous data table field set which is preconfigured for the second data table in the first data table. And under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table. Based on the second data table operation, the second data table is updated. In this way, the data table operation notification is triggered when the first data table operation is written into the message queue, so that the real-time performance of data synchronization can be improved. If the operation data table field included in the first data table operation belongs to the synchronous data table field set, which indicates that the second data table needs to synchronize the operation data table field in the first data table, the first data table operation is converted into the second data table operation for the second data table according to the database operation conversion rule for the first data table operation, and the second data table is updated based on the second data table operation, so that the efficiency of data synchronization between heterogeneous databases can be effectively improved.

Drawings

FIG. 1 is an application environment diagram of a data synchronization method in one embodiment;

FIG. 2 is a flow chart of a data synchronization method in one embodiment;

FIG. 3 is a flow diagram of a process for acquiring a set of synchronization data table fields in one embodiment;

FIG. 4 is a flow chart of a data synchronization method according to another embodiment;

FIG. 5 is a block diagram of a data synchronization device in one embodiment;

FIG. 6 is a block diagram of a data synchronization device in another embodiment;

FIG. 7 is an internal block diagram of a computer device in one embodiment;

fig. 8 is an internal structural view of a computer device in another embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

The data synchronization method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, which may be smart televisions, smart car devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers. The terminal 102 and the server 104 may be directly or indirectly connected through wired or wireless communication, which is not limited herein.

Both the terminal and the server can be used separately to execute the data synchronization method provided in the embodiments of the present application.

For example, the terminal acquires a data table operation notification sent by a message queue subscribed by the second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases. And the terminal acquires a synchronous data table field set which is preconfigured for the second data table in the first data table. Under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, the terminal converts the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table; based on the second data table operation, the second data table is updated.

The terminal and the server may also cooperate to perform the data synchronization method provided in the embodiments of the present application.

For example, the terminal sends a data table operation notification sent by a message queue subscribed by the second data table to the server; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases. The server obtains a set of synchronization data table fields in the first data table that are preconfigured for the second data table. Under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, the server converts the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table; and updating the second data table based on the second data table operation to obtain a second data table synchronization result. The server sends the second data table synchronization result to the terminal, and the terminal can display the second data table synchronization result.

In one embodiment, as shown in fig. 2, a data synchronization method is provided, and the method is applied to a computer device, which is a terminal or a server, and is executed by the terminal or the server, or may be implemented through interaction between the terminal and the server. The data synchronization method comprises the following steps:

step S202, a data table operation notice sent by a message queue subscribed by a second data table is obtained; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases.

Wherein the second data table is a data table which needs to be data synchronized with the first data table. The first data table operation refers to a database statement for altering data in the first data table. The message queue for the second data table subscription refers to a message queue for receiving first data table operations for the first data table. The data table operation notification refers to a notification carrying the first data table operation, which is triggered and generated by writing the first data table operation into the message queue, and is used for notifying the second data table to perform data synchronization. The first database and the second database are heterogeneous databases, i.e. databases of different types, and may be located on the same device or may be located on different devices respectively.

The computer device, upon collecting a first data table operation for the first data table in the first database, writes the first data table operation to a message queue of the second data table subscription. And triggering the message queue to send a data table operation notification when the first data table operation is written into the message queue. The computer equipment acquires a data table operation notification carrying a first data table operation aiming at the first data table sent by a message queue subscribed by the second data table, and performs data synchronization on the second data table based on the first data table operation.

Step S204, a synchronous data table field set preconfigured for a second data table in the first data table is obtained.

Wherein the data table field refers to each field contained in the data table. The synchronous data table field refers to a data table field which is preconfigured for the second data table and needs to be synchronized for the second data table in the first data table. A set of synchronized data table fields refers to a set that includes the individual data table fields in the first data table that the second data table needs to be synchronized. For example, when the first data table includes data table fields of employee name, job number, age, sex, job position, department, etc., and the second data table is a data table including employee name, job position, department generated based on the first data table, then the set of synchronous data table fields preconfigured for the second data table in the first data table is a set of three data table fields of employee name, job position, and department.

Illustratively, upon receiving a data table operation notification carrying a first data table operation, the computer device obtains a set of synchronized data table fields including respective data table fields in the first data table for which synchronization is required for the second data table.

In step S206, when the operation data table field included in the first data table operation belongs to the synchronous data table field set, the first data table operation is converted based on the database operation conversion rule for the first data table operation, so as to obtain a second data table operation for the second data table.

The data table field of the first data table operation is a data table field in the first data table for which the first data table operation is aimed, that is, a data table field to which the data modified in the first data table by the first data table operation belongs. For example, if the first data table operation is a delete operation for the age data corresponding to employee a, the operation data table field corresponding to the first data table operation is the age field in the first data table.

The first data table and the second data table are different types of data tables, and the second data table has difficulty in resolving the first data table operation for the first data table, so that the first data table operation needs to be converted into a data table operation capable of acting on the second data table when data synchronization is performed. When the data table field in the second data table is the same as the corresponding synchronous data table field in the first data table, that is, when the field values corresponding to the fields are completely the same, only the first data table operation is required to be converted into the second data table operation, the data table field values corresponding to the operation data table fields carried by the first data table operation are not required to be converted, and if the data table field in the second data table is obtained by carrying out certain conversion on the basis of the corresponding synchronous data table field in the first data table, the data table field values corresponding to the operation data table fields carried by the first data table operation are required to be converted while the first data table operation is converted into the second data table operation.

For example, when the first data table includes data table fields such as employee name, employee number, age, job position, department, etc., the second data table is a data table generated based on data table fields such as employee name and employee number in the first data table, and the second data table includes data table fields such as employee surname and employee number, etc., where the data table field value of the employee surname is obtained by converting the data table field value based on the employee name in the first data table, that is, the employee surname is extracted from the employee name, and the data table field value corresponding to the employee surname is obtained. If the operation data table field carried by the first data table operation is the employee surname, converting the first data table operation into the second data table operation corresponding to the second data table, and correspondingly converting the data table field value corresponding to the employee name carried by the first data table operation to obtain the data table field value corresponding to the employee surname. For the staff work number in the second data table, the data table field value corresponding to the staff work number in the first data table is directly used as the data table field value corresponding to the staff work number in the second data table, and the data table field value of the staff work number is not required to be converted. If the operation data table field carried by the first data table operation is the employee work number, only the first data table operation is required to be converted into the second data table operation, and the data table field value corresponding to the employee work number carried by the first data table operation is not required to be converted.

The database operation conversion rule refers to a rule which is preconfigured and used for indicating how to convert the first data table operation into the second data table operation aiming at the second data table, and specifically, each synchronous data table field in the synchronous data table field set respectively has a corresponding database operation conversion rule. For example, if the synchronous data table field set includes data table fields such as employee name, job position, age, etc., the second data table sets corresponding database operation conversion rules for the synchronous data table fields such as employee name, job position, age, etc. in the first data table.

Illustratively, the computer device compares an operational data table field contained by the first data table operation with each of the set of synchronization data table fields. And when the operation data table field contained in the first data table operation belongs to the synchronous data table field set, acquiring a database operation conversion rule aiming at the operation data table field contained in the first data operation, and converting the first data table operation into a second data table operation aiming at the second data table based on the database operation conversion rule.

Step S208, based on the second data table operation, updating the second data table.

The computer device, based on the second data table operation, updates the data table field value of the data table field to be synchronized corresponding to the data table field to be synchronized in the data table of the operation included in the first data table operation.

In the data synchronization method, the data table operation notification sent by the message queue subscribed by the second data table is obtained; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases. And acquiring a synchronous data table field set which is preconfigured for the second data table in the first data table. And under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table. Based on the second data table operation, the second data table is updated. The method belongs to the technical field of big data, and can improve the real-time performance of data synchronization by triggering the data table operation notification when the first data table operation is written into the message queue. If the operation data table field included in the first data table operation belongs to the synchronous data table field set, which indicates that the second data table needs to synchronize the operation data table field in the first data table, the first data table operation is converted into the second data table operation for the second data table according to the database operation conversion rule for the first data table operation, and the second data table is updated based on the second data table operation, so that the efficiency of data synchronization between heterogeneous databases can be effectively improved.

In one embodiment, as shown in fig. 3, obtaining a set of synchronization data table fields preconfigured for a second data table in a first data table includes:

in step S302, when the data synchronization type between the first data table and the second data table is the global synchronization type, each data table field in the first data table is used as a synchronization data table field, and a synchronization data table field set preconfigured for the second data table in the first data table is obtained based on each synchronization data table field.

Step S304, when the data synchronization type between the first data table and the second data table is the local synchronization type, a plurality of data table fields to be synchronized for the first data table in the second data table are obtained, the data table fields corresponding to each data table field to be synchronized in the first data table are used as the synchronization data table fields, and a synchronization data table field set preconfigured for the second data table in the first data table is obtained based on each synchronization data table field.

The data synchronization type refers to a type of data synchronization between the first data table and the second data table. The global synchronization type refers to that data table fields in the first data table and data table fields in the second data table are in one-to-one correspondence, and the second data table needs to synchronize all data table fields in the first data table. The local sync type means that the second data table only needs to synchronize part of the data table fields in the first data table. The data table field to be synchronized refers to a data table field in the second data table, which needs to be synchronized with the first data table.

For example, if the data table field included in the first data table is an employee name, a job number, and a job position, the second data table is the same data table as the first data table generated based on the first data table, that is, the data table copy of the first data table, and the data table field included in the second data table is also an employee name, a job number, and a job position, then the data synchronization type between the first data table and the second data table is a global synchronization type, and each data table field included in the second data table is a field to be synchronized for the first data table in the second data table. If the data table fields included in the first data table are employee names, job numbers, ages, sexes, positions, departments, the second data table is a data table generated based on the data table fields including employee names, positions, departments, and the like in the first data table, and in addition, the second data table also includes the data table fields including employee ages, academies, remaining years, and the like, the data synchronization type between the first data table and the second data table is a local synchronization type, the synchronization data table field set preconfigured for the second data table in the first data table is a set composed of the synchronization data table fields including employee names, positions, departments, and the like in the first data table, and the data table fields including employee names, positions, departments, and the like in the second data table are the data table fields to be synchronized for the first data table in the second data table.

When the data synchronization type between the first data table and the second data table is a global synchronization type, it is illustrated that the data table fields in the first data table and the data table fields in the second data table are in one-to-one correspondence, the second data table needs to synchronize all the data table fields in the first data table, the computer device takes each data table field contained in the first data table as a synchronization data table field, and obtains a synchronization data table field set preconfigured for the second data table in the first data table based on each synchronization data table field.

When the data synchronization type between the first data table and the second data table is the local synchronization type, the second data table is only required to synchronize part of the data table fields in the first data table, the computer equipment acquires each data table field to be synchronized of the second data table aiming at the first data table, acquires the corresponding data table field of the data table field to be synchronized in the first data table as the synchronization data table field based on the mapping relation between the data table field of the first data table and the data table field of the second data table, and acquires the synchronization data table field set pre-configured aiming at the second data table in the first data table based on each synchronization data table field.

In the above embodiment, when the global synchronization type is between the first data table and the second data table, each data field in the first data table is used as a synchronization data table field, so as to obtain a synchronization data table field set. When the local synchronization type exists between the first data table and the second data table, each data table field, which needs to be synchronized, of the second data table in the first data table is used as a synchronization data table field, and a synchronization data table field set is obtained. Therefore, for different data synchronization types, the synchronous data table field set can be rapidly determined, and based on the synchronous data table field set, whether the first data table operation in the data table operation notification is the data table operation requiring synchronization of the second data table can be rapidly and accurately judged, so that the data synchronization efficiency is improved.

In one embodiment, the data synchronization method further comprises:

acquiring an operation type corresponding to the first data table operation under the condition that the operation data table field contained in the first data table operation does not belong to the synchronous data table field set; when the operation type corresponding to the first data table operation is a field addition type and the synchronous operation type set of the second data table aiming at the first data table comprises the field addition type, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a third data table operation aiming at the second data table; the second data table is updated based on the third data table operation.

The operation type is a type of data table operation, for example, the data table operation can be added, deleted or changed. The synchronization operation type set is a set of operation types that are preconfigured for the second data table and need to be synchronized with the first data table, for example, the synchronization operation type set may include deletion and modification, and at this time, if the first data table operation is of an increment type, the first data table operation is discarded, data synchronization for the first data table operation belonging to the increment type is not needed, if the first data table operation is of a modification type, it is determined whether an operation data table field included in the first data table operation belongs to the synchronization data table set, and if the operation data table field belongs to the synchronization data table set, data synchronization for the first data table operation is performed; the set of synchronization operation types may also include three types of addition, deletion, and modification, where data synchronization for the first data table operation belonging to the addition type is required. The field addition type refers to the operation type of the newly added field in the data table. The third data table operation refers to the data table operation of the field adding type aiming at the second data table, which is obtained by converting the first data table operation of the field adding type based on the database operation conversion rule corresponding to the first data table operation of the field adding type when the first data table operation is of the field adding type.

For example, since the newly added data table field is not always in the synchronous data table field, when the operand data table field included in the first data table operation does not belong to the synchronous data table field set, the computer device obtains the operation type corresponding to the first data table operation, and when the operation type is the field addition type, the computer device obtains the synchronous operation type set of the second data table for the first data table. When the field addition type is included in the synchronous operation type set, a pre-configured database operation conversion rule for indicating how to convert a first data table operation of the field addition type into a second data table operation of a second data table is obtained, the first data table operation is converted into a third data table operation of the second data table based on the database operation conversion rule corresponding to the field addition type, and fields corresponding to data table fields of an operation included in the first data table operation are added in the second data table based on the third data table operation.

In the above embodiment, when the operation data table field included in the first data table operation does not belong to the synchronous data table field set, the operation type corresponding to the first data table operation is obtained, when the operation type is the field addition type, whether the synchronous operation type set of the second data table for the first data table includes the field addition type is further determined, and if the synchronous operation type set includes the field addition type, data synchronization for the first data table operation is performed. In this way, the accuracy of data synchronization can be improved, and meanwhile, the flexibility of data synchronization can be improved by setting the synchronous operation type set of the second data table aiming at the first data table.

In one embodiment, converting a first data table operation based on database operation conversion rules for the first data table operation to obtain a second data table operation for a second data table, includes:

acquiring an initial configuration field value set corresponding to the operation of a first data table; the initial configuration field value set comprises field values respectively corresponding to operation configuration fields in the first data table operation; acquiring field value conversion information and a second data table operation template contained in a data table operation conversion rule aiming at the first data table operation; and filling the second data table operation template based on the field value conversion information and the initial configuration field value set to obtain a second data table operation aiming at the second data table.

Wherein the operation configuration field is each field included in the operation of the data table. For example, the operation configuration field may be a data table identifier, a data table field identifier, a field corresponding to a field value corresponding to an operation table field, and so on. The initial configuration field value refers to a field value corresponding to an operation configuration field to be configured, which is included in the first data table operation, and is used for indicating how to change the data table. The initial configuration field value set refers to a set including field values corresponding to operation configuration fields to be configured in the first data table operation.

The field value conversion information is used for indicating how to convert the field value of the operation configuration field contained in the first data table operation into the field value corresponding to the corresponding configuration field in the second data table operation. The second data table operation template refers to a template of a database statement corresponding to the second database, and a plurality of fields needing to be configured contained in the second data table operation template are filled to obtain the database statement capable of acting on the second database. The second data table operation template included in the data table operation conversion rule for the first data table operation refers to a second data table operation template corresponding to the operation type to which the first data table operation belongs in the second database, for example, when the first data table operation is that the corresponding operation type is a field addition type, the second data table operation template for the first data table operation is that a field addition type corresponds to a database statement in the second database.

The computer device analyzes the first data table operation to obtain initial configuration field values corresponding to the operation configuration fields in the first data table operation. The data table operation conversion rule may include field value conversion information and a second data table operation template corresponding to the plurality of operation types, respectively. The operation type of the first data table operation is obtained, and field value conversion information and a second data table operation template corresponding to the operation type are obtained from a data table operation conversion rule corresponding to the first data table operation. For example, the set database operation conversion rule of the second data table for the data table field of the age in the first data table may include field value conversion information and a second data table operation template corresponding to the field deletion type and the field change type respectively, and if the first data table operation is a deletion operation for the age data corresponding to the employee a, the operation type to which the first data table operation belongs is the field deletion type, at this time, the field value conversion information and the second data table operation template corresponding to the field deletion type are obtained from the database operation conversion rule set by the second data table for the data table field of the age in the first data table.

And converting the initial configuration field value set according to the field value conversion information to obtain a target configuration field value set, and filling a second data table operation template based on the target configuration field value set to obtain a second data table operation corresponding to the first data table operation and aiming at the second data table.

In the above embodiment, by acquiring the initial configuration field value set corresponding to the first data table operation, converting the initial configuration field value set based on the field value conversion information included in the data table operation conversion rule corresponding to the first data table operation to obtain the target configuration field value set, and filling the target configuration field value set into the second data table operation template to obtain the second data table operation, the first data table operation can be quickly and accurately converted into the second data table operation for the second data table, and the efficiency of data synchronization is improved.

In one embodiment, filling the second data table operation template based on the field value conversion information and the initial configuration field value set to obtain a second data table operation for the second data table comprises:

when the data table field value conversion information contained in the field value conversion information is empty, filling the target configuration field value set into a second data table operation template to obtain a second data table operation aiming at a second data table; and when the data table field value conversion information contained in the field value conversion information is not empty, converting the initial configuration field value set based on the data table field value conversion information and the basic field value conversion information to obtain a target configuration field value set, and filling the target configuration field value set into a second data table operation template to obtain a second data table operation aiming at the second data table.

The data table field value conversion information is used for indicating how to convert a data table field value corresponding to an operation data table field carried in a first data table operation into a data table field value corresponding to a corresponding data table field to be synchronized in a second data table, and may be a formula or a function for converting the data table field value carried in the data table operation into a field value corresponding to the corresponding data table field to be synchronized in the second data table, where the function may be a basic function or a custom function, and the custom function is a code block defined in an entity class. The basic field value conversion information is used for indicating how to convert field values corresponding to each operation configuration field except for the data table field values corresponding to the operation table fields carried in the first data table operation into field values corresponding to the corresponding operation configuration fields in the second data table operation. Specifically, the basic field value conversion information may include conversion information for converting a data table identifier of a first data table in the initial configuration field set into a data table identifier of a second data table, conversion information for converting a field identifier of an operation data table field in the initial configuration field set into a field identifier of a corresponding data table field to be synchronized in the second data table, and the like.

The target configuration field value set refers to a configuration field value set obtained by converting initial configuration field values in the initial configuration field set based on field value conversion information.

When the data table field value conversion information included in the field value conversion information is empty, converting the field values corresponding to the operation configuration fields, except for the data table field values corresponding to the operation configuration fields, carried in the first data table operation, into the field values corresponding to the corresponding operation configuration fields in the second data table operation based on the basic field value conversion information included in the field value conversion information. And taking the data table field value corresponding to the operation data table field as the data table field value corresponding to the corresponding data table field to be synchronized in the second data table. And obtaining a target configuration field value set based on the field value of the data table corresponding to the data table field to be synchronized and the field value corresponding to each operation configuration field in the second data table operation. And filling each field value in the target configuration field value set into a corresponding operation configuration field in the second data table operation template respectively to obtain second data table operation aiming at the second data table.

When the data table field value conversion information contained in the field value conversion information is not null, converting the data table field value corresponding to the data table field to be synchronized carried in the first data table operation into the data table field value corresponding to the corresponding data table field to be synchronized in the second data table based on the data table field value conversion information, and converting the field values corresponding to the operation configuration fields carried in the first data table operation except the data table field value corresponding to the data table field to be synchronized into the field values corresponding to the corresponding operation configuration fields in the second data table operation based on the basic field value conversion information contained in the field value conversion information. And obtaining a target configuration field value set based on the field value of the data table corresponding to the data table field to be synchronized and the field value corresponding to each operation configuration field in the second data table operation. And filling each field value in the target configuration field value set into a corresponding operation configuration field in the second data table operation template respectively to obtain second data table operation aiming at the second data table.

In the above embodiment, the initial configuration field set is converted based on the data table field value conversion information and the base field value conversion information included in the field value conversion information, to obtain the target configuration field value set. Based on the data table field value conversion information and the basic field value conversion information contained in the field value conversion information, the initial configuration field value set can be quickly and accurately converted into the target configuration field value set aiming at the second data table operation. And filling the target configuration field value set into a second data table operation template to obtain a second data table operation aiming at the second data table. Therefore, the operation of the first data table is quickly converted into the operation of the second data table aiming at the second data table, and the efficiency of data synchronization can be effectively improved.

In one embodiment, the data synchronization method further comprises:

when the current data synchronization period is finished, acquiring each data sequence of the operation time in the current data synchronization period in the first data table; based on each data sequence, the corresponding data sequence in the second data table is updated.

The data synchronization period refers to a preset data synchronization period, the data synchronization period can be set according to actual needs, and in an actual implementation process, the data synchronization period can be set to be one day. The current data synchronization period refers to a data synchronization period to which the current time belongs. A data sequence refers to a sequence of a plurality of data table field values in a first data table. In an actual implementation process, the data sequence may be a sequence composed of field values corresponding to a certain field, that is, a column of data corresponding to a certain field in the data table, for example, a field value sequence corresponding to a field of the data table, that is, an employee name, is a data sequence, which includes each employee name. The data sequence may also be a record in the data table, that is, a row of data in the data table, including data table field values corresponding to an entity in the data table under each data table field, for example, a sequence formed by field values corresponding to employee a under each data table field such as employee name, job number, job position, etc. may be a data sequence. The operation time is the time when the update of the data sequence occurred last time.

In an exemplary embodiment, at the end of the current data synchronization period, operation times corresponding to the data sequences in the first data table are obtained, and the corresponding data sequence in the second data table is updated based on the data sequences of the operation times in the current data synchronization period.

In the above embodiment, since the message may be lost, when each data synchronization period is finished, the second data table is subjected to data synchronization once based on the data sequence changed in the current data synchronization period in the first data table, so that the loss caused by data loss can be reduced, and the accuracy of data synchronization can be improved. When the current data synchronization period is finished, taking the data sequences comprising a plurality of data units as units, counting the operation time corresponding to each data sequence respectively to determine to generate a changed data sequence, and not counting the operation time corresponding to each data unit in the first data table respectively, so that the computer resource can be saved, and the data synchronization efficiency can be improved.

In a specific embodiment, the data synchronization method provided by the application can be applied to a data synchronization system of a heterogeneous database, as shown in fig. 4, where the data synchronization system includes a data acquisition module, a message module, and a data synchronization module. The data synchronization method comprises the following steps:

1. Data acquisition module

When processing online business, a data acquisition module in the data synchronization system acquires actions such as adding, deleting, changing and the like aiming at a first data table, when acquiring a first database operation, the first database operation is assembled into a message entity, a TDMQ (Tencent Distributed Message Queue) cluster is adapted, and the message entity is released to a message queue between a second data table and the first data table.

2. Message module

And a message module in the data synchronization system monitors the message queue in real time, and sends a data table operation notification carrying the message entity corresponding to the first database operation after monitoring that the message entity is newly added to the message queue. After the message is successfully forwarded, changing the message recording state into success, if the transmission fails, recording the state as failure, and carrying out retransmission retry later, wherein the number of times is configurable.

3. Data synchronization module

When receiving the data table operation notification sent by the message module, a data synchronization module in the data synchronization system acquires a synchronization data table field set which is preconfigured for a second data table in the first data table. And under the condition that the operation data table field contained in the first data table operation belongs to the synchronous data table field set, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at a second data table. Based on the second data table operation, the second data table is updated. And under the condition that the operation field contained in the first data table operation does not belong to the synchronous data table field set, acquiring the operation type corresponding to the first data table operation. When the operation type corresponding to the first data table operation is a field addition type and the synchronous operation type set of the second data table aiming at the first data table comprises a new addition type, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a second data table operation aiming at the second data table. Based on the second data table operation, the second data table is updated. And (4) carrying out full synchronization at night according to the corresponding relation between the configured data tables because the messages can be lost.

In the above embodiment, a message queue mechanism is used to establish a data synchronization system, and first, actions such as adding, deleting, changing, etc. of a database table in an online service are collected, and collected information is sent. The message module monitors the appointed message channel and records and forwards the message. The data synchronization module receives the message notification of the message module, and performs data synchronization according to the corresponding relation of the table and the table field between the configured heterogeneous databases. The interface asynchronous processing mechanism is adopted to reduce the influence of database synchronization on online service, and meanwhile, the message queue is adopted to ensure the accuracy and the instantaneity of data synchronization, and a batch compensation mechanism is also arranged for ensuring the data integrity. The problem of data synchronization between heterogeneous databases is solved.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data synchronization device for realizing the above related data synchronization method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data synchronization device provided below may refer to the limitation of the data synchronization method hereinabove, and will not be repeated herein.

In one embodiment, as shown in fig. 5, there is provided a data synchronization apparatus including: an operation notification acquisition module 502, a synchronization field set acquisition module 504, a data table operation conversion module 506, and a data table update module 508, wherein:

an operation notification obtaining module 502, configured to obtain a data table operation notification sent by a message queue subscribed by the second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into a message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases.

The synchronization field set obtaining module 504 is configured to obtain a synchronization data table field set preconfigured for the second data table in the first data table.

The data table operation conversion module 506 is configured to convert the first data table operation based on a database operation conversion rule for the first data table operation to obtain a second data table operation for the second data table when the operation data table field included in the first data table operation belongs to the synchronous data table field set.

The data table updating module 508 is configured to update the second data table based on the second data table operation.

In one embodiment, the sync field set acquisition module 504 is further configured to:

when the data synchronization type between the first data table and the second data table is the global synchronization type, respectively taking each data table field in the first data table as a synchronization data table field, and obtaining a synchronization data table field set which is preconfigured for the second data table in the first data table based on each synchronization data table field; when the data synchronization type between the first data table and the second data table is the local synchronization type, acquiring a plurality of data table fields to be synchronized for the first data table in the second data table, taking the data table fields corresponding to the data table fields to be synchronized in the first data table respectively as the synchronization data table fields, and acquiring a synchronization data table field set preconfigured for the second data table in the first data table based on the synchronization data table fields.

In one embodiment, the data table operation conversion module 506 is further configured to:

when the data table field value conversion information contained in the field value conversion information is empty, converting the initial configuration field set based on the basic field value conversion information contained in the field value conversion information to obtain a target configuration field value set, and filling the target configuration field value set into a second data table operation template to obtain a second data table operation aiming at a second data table; and when the data table field value conversion information contained in the field value conversion information is not empty, converting the initial configuration field value set based on the data table field value conversion information and the basic field value conversion information to obtain a target configuration field value set, and filling the target configuration field value set into a second data table operation template to obtain a second data table operation aiming at the second data table.

In one embodiment, as shown in fig. 6, the data synchronization device further includes:

the data table updating module 602 is configured to obtain an operation type corresponding to the first data table operation when the operation data table field included in the first data table operation does not belong to the synchronous data table field set; when the operation type corresponding to the first data table operation is a field addition type and the synchronous operation type set of the second data table aiming at the first data table comprises the field addition type, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a third data table operation aiming at the second data table; the second data table is updated based on the third data table operation.

A data sequence synchronization module 604, configured to acquire, at the end of the current data synchronization period, each data sequence of the first data table having an operation time within the current data synchronization period; based on each data sequence, the corresponding data sequence in the second data table is updated.

According to the data synchronization device, the data synchronization instantaneity can be improved by triggering the data table operation notification when the first data table operation is written into the message queue. If the operation data table field included in the first data table operation belongs to the synchronous data table field set, which indicates that the second data table needs to synchronize the operation data table field in the first data table, the first data table operation is converted into the second data table operation for the second data table according to the database operation conversion rule for the first data table operation, and the second data table is updated based on the second data table operation, so that the efficiency of data synchronization between heterogeneous databases can be effectively improved.

The modules in the data synchronization device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data such as a first data table operation, a synchronous data table field set and the like. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data synchronization method.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 8. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data synchronization method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structures shown in fig. 7 and 8 are block diagrams of only some of the structures that are relevant to the present application and are not intended to limit the computer device on which the present application may be implemented, and that a particular computer device may include more or fewer components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the method embodiments described above.

In one embodiment, a computer program product or computer program is provided that includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-described method embodiments.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method of data synchronization, the method comprising:

acquiring a data table operation notification sent by a message queue subscribed by a second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into the message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases;

Acquiring a synchronous data table field set which is preconfigured in the first data table aiming at the second data table;

converting the first data table operation based on a database operation conversion rule aiming at the first data table operation under the condition that an operation data table field contained in the first data table operation belongs to the synchronous data table field set, so as to obtain a second data table operation aiming at the second data table;

updating the second data table based on the second data table operation.

2. The method of claim 1, wherein the obtaining the set of synchronization data table fields of the first data table that are preconfigured for the second data table comprises:

when the data synchronization type between the first data table and the second data table is a global synchronization type, respectively taking each data table field in the first data table as a synchronization data table field, and obtaining a synchronization data table field set which is preset for the second data table in the first data table based on each synchronization data table field;

when the data synchronization type between the first data table and the second data table is a local synchronization type, acquiring a plurality of data table fields to be synchronized for the first data table in the second data table, taking data table fields corresponding to each data table field to be synchronized in the first data table as synchronization data table fields, and acquiring a synchronization data table field set preconfigured for the second data table in the first data table based on each synchronization data table field.

3. The method according to claim 1, wherein the method further comprises:

acquiring an operation type corresponding to the first data table operation under the condition that an operation data table field contained in the first data table operation does not belong to the synchronous data table field set;

when the operation type corresponding to the first data table operation is a field addition type and the synchronous operation type set of the second data table aiming at the first data table comprises the field addition type, converting the first data table operation based on a database operation conversion rule aiming at the first data table operation to obtain a third data table operation aiming at the second data table;

updating the second data table based on the third data table operation.

4. The method of claim 1, wherein converting the first data table operation based on database operation conversion rules for the first data table operation to obtain a second data table operation for the second data table, comprises:

acquiring an initial configuration field value set corresponding to the first data table operation; the initial configuration field value set comprises field values respectively corresponding to operation configuration fields in the first data table operation;

Acquiring field value conversion information and a second data table operation template contained in a data table operation conversion rule aiming at the first data table operation;

and filling the second data table operation template based on the field value conversion information and the initial configuration field value set to obtain a second data table operation aiming at the second data table.

5. The method of claim 4, wherein the populating the second data table operation template based on the field value conversion information and the initial set of configuration field values to obtain a second data table operation for the second data table comprises:

when the data table field value conversion information contained in the field value conversion information is empty, converting the initial configuration field set based on the basic field value conversion information contained in the field value conversion information to obtain a target configuration field value set, and filling the target configuration field value set into the second data table operation template to obtain a second data table operation aiming at the second data table;

and when the data table field value conversion information contained in the field value conversion information is not empty, converting the initial configuration field value set based on the data table field value conversion information and the basic field value conversion information to obtain a target configuration field value set, and filling the target configuration field value set into the second data table operation template to obtain a second data table operation aiming at the second data table.

6. The method according to claim 1, wherein the method further comprises:

when the current data synchronization period is finished, acquiring each data sequence of the operation time in the current data synchronization period in the first data table;

and updating the corresponding data sequences in the second data table based on the data sequences.

7. A data synchronization device, the device comprising:

the operation notification acquisition module is used for acquiring a data table operation notification sent by a message queue subscribed by the second data table; the data table operation notification carries a first data table operation aiming at a first data table, and the data table operation notification is triggered by writing the first data table operation into the message queue; the first database to which the first data table belongs and the second database to which the second data table belongs are heterogeneous databases;

a synchronous field set acquisition module, configured to acquire a synchronous data table field set preconfigured for the second data table in the first data table;

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.