CN111367924A - Data synchronization method and device, electronic equipment and storage medium - Google Patents
Data synchronization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111367924A CN111367924A CN202010121734.0A CN202010121734A CN111367924A CN 111367924 A CN111367924 A CN 111367924A CN 202010121734 A CN202010121734 A CN 202010121734A CN 111367924 A CN111367924 A CN 111367924A
- Authority
- CN
- China
- Prior art keywords
- data table
- synchronization
- data
- structure information
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000001360 synchronised effect Effects 0.000 claims abstract description 41
- 238000012544 monitoring process Methods 0.000 claims abstract description 36
- 230000008569 process Effects 0.000 claims abstract description 29
- 230000008859 change Effects 0.000 claims abstract description 19
- 230000004048 modification Effects 0.000 claims description 26
- 238000012986 modification Methods 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012423 maintenance Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 238000011217 control strategy Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The specification provides a data synchronization method, a data synchronization device, an electronic device and a storage medium, wherein the method comprises the following steps: monitoring table structure information of a source data table in the process of executing a synchronization task, wherein the synchronization task is used for implementing data synchronization operation from the source data table to a target data table; if the table structure information changes, modifying the task statement of the synchronous task according to the changed table structure information; and executing the modified synchronization task to continue to implement the data synchronization operation from the source data table to the destination data table. Therefore, the change of the table structure information of the source data table is monitored, the table structure information of the synchronous task and/or the target data table is adaptively modified, the synchronous task is ensured not to be interrupted and executed due to the change of the table structure information of the source data table, and the effective execution of the synchronous task is further ensured.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data synchronization method, an apparatus, an electronic device, and a storage medium.
Background
At present, in the process of synchronizing data in a source data table to a destination data table, if table structure information of the source data table changes, for example, fields are added to the source data table, fields in the source data table are deleted, field names of the source data table are modified, field types and lengths of the source data table are modified, so that the table structure information of the source data table is not matched with a task statement of an executing synchronization task or structure information of the destination data table, and the synchronization task fails.
Disclosure of Invention
To overcome the problems in the related art, the present specification provides a data synchronization method, apparatus, electronic device, and storage medium.
According to a first aspect of embodiments of the present specification, there is provided a data synchronization method including:
monitoring table structure information of a source data table in the process of executing a synchronization task, wherein the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
if the table structure information changes, modifying the task statement of the synchronous task according to the changed table structure information;
and executing the modified synchronization task to continue to implement the data synchronization operation from the source data table to the destination data table.
Optionally, if the table structure information changes, the data synchronization method further includes:
updating the target data table according to the changed table structure information;
implementing a data synchronization operation of the source data table to the destination data table, including:
and implementing the data synchronization operation of the source data table to the updated destination data table.
Optionally, updating the destination data table includes:
correspondingly modifying the table structure information of the target data table according to the changed table structure information;
or, creating a new destination data table, wherein the table structure information of the new destination data table is the same as the changed table structure information.
Optionally, if the table structure information changes, the data synchronization method further includes:
and modifying all other tasks which are not executed and completed and are related to the source data table according to the changed table structure information.
Optionally, the other tasks that are not performed for completion include:
a synchronization task for implementing data synchronization operation of the source data table to other destination data tables;
or, a synchronization task for implementing data synchronization operation of the destination data table to other destination data tables.
Optionally, when the synchronization task is used to implement data synchronization operation from multiple source data tables to one destination data table, if the table structure information changes and includes that the associated fields of the multiple source data tables change, the data synchronization method further includes:
and stopping executing the synchronous task and generating alarm information.
Optionally, the source data table is stored in a source database;
monitoring table structure information of a source data table, including:
acquiring a DDL (Data Definition Language) log of the source database;
and monitoring the table structure information of the source data table according to the DDL operation information recorded in the DDL log.
According to a second aspect of embodiments of the present specification, there is provided another data synchronization method including:
monitoring table structure information of a source data table in the process of executing a synchronization task, wherein the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
if the table structure information changes, updating the table structure information of the target data table according to the changed table structure information;
and implementing the data synchronization operation of the source data table to the updated destination data table.
According to a third aspect of embodiments herein, there is provided a data synchronization apparatus comprising:
the system comprises a monitoring module, a synchronization module and a processing module, wherein the monitoring module is used for monitoring the table structure information of a source data table in the process of executing a synchronization task, and the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
the modification module is used for modifying the task statement of the synchronous task according to the changed table structure information under the condition that the table structure information is changed;
and the execution module is used for executing the modified synchronization task so as to continuously implement the data synchronization operation from the source data table to the destination data table.
According to a fourth aspect of embodiments herein, there is provided another data synchronization apparatus, including:
the system comprises a monitoring module, a synchronization module and a processing module, wherein the monitoring module is used for monitoring the table structure information of a source data table in the process of executing a synchronization task, and the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
the modification module is used for updating the table structure information of the target data table according to the changed table structure information under the condition that the table structure information is changed;
and the execution module is used for implementing the data synchronization operation of the source data table to the updated target data table.
According to a fifth aspect of embodiments of the present specification, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the data synchronization method of any one of the above items when executing the computer program.
According to a sixth aspect of embodiments herein, there is provided a computer readable storage medium, having stored thereon a computer program which, when executed by a processor, implements the steps of the data synchronization method of any of the above.
The technical scheme provided by the embodiment of the specification can have the following beneficial effects: by monitoring the change of the table structure information of the source data table, the table structure information of the synchronous task and/or the target data table is adaptively modified, the synchronous task is ensured not to be interrupted and executed due to the change of the table structure information of the source data table, and the effective execution of the synchronous task is further ensured.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the specification.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present specification and together with the description, serve to explain the principles of the specification.
FIG. 1 is a flow chart illustrating a method of data synchronization according to an exemplary embodiment of the present description.
FIG. 2 is a flow chart illustrating another method of data synchronization shown in accordance with an exemplary embodiment of the present description.
Fig. 3 is a block diagram of a data synchronization apparatus according to an exemplary embodiment.
Fig. 4 is a schematic structural diagram of an electronic device shown in the present specification according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the specification, as detailed in the appended claims.
The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present specification. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
In the related art, in the process of synchronizing data in a source data table to a destination data table, if table structure information of the source data table changes, for example, a field is added to the source data table, a field in the source data table is deleted, a field name of the source data table is modified, a field length or type of the source data table is modified, and the like, the table structure information of the source data table may not match a task statement of an executing synchronization task or structure information of the destination data table, which may cause interruption of execution of the synchronization task, and may not complete data synchronization operation.
Based on this, embodiments of the present invention provide a data synchronization method, which is used for automatically modifying a synchronization statement of a synchronization task and/or table structure information of a destination data table in a data synchronization process when the table structure information of a source data table changes, so as to effectively complete the synchronization task. The table structure information includes a database name for storing the data table, a data table name, a field length, a field type, a field cleaning rule, and the like.
The user can create a sync task using, but not limited to, an ETL tool (a tool that extracts, converts, and loads data from a source to a destination), and the sync task is established according to the source data table name and its full or partial field name, the destination data table name and its full or partial field name provided by the user. The ETL tool may contain a plurality of synchronization tasks, and the ETL synchronizes data obtained from a corresponding source data table into a corresponding destination data table by performing each synchronization task.
For ease of understanding, table structure information changes of the source data table are classified into 2 types, and the first type of changes relates to changes of field level table structures, including: deletion of fields, addition of fields, modification of field names, etc.; the second type of change involves changes to field attributes, including: a change in field type, a change in field length, a change in field cleansing rules, etc.
Next, the data synchronization process of the embodiment of the present specification is described in detail with respect to a first variation of the source data table.
As shown in fig. 1, fig. 1 is a flow chart illustrating a data synchronization method according to an exemplary embodiment, including the following steps:
The synchronization task is used for implementing data synchronization operation from the source data table to the destination data table. The source data table is stored in a source database, and the destination data table is stored in a destination database. The source database and the destination database may be the same type of database or different types of databases.
If the table structure information of the Data table in the database is modified, there is an ALTER (table structure modification) related record in a DDL (Data definitional language) log. Therefore, by acquiring the ALTER record information (DDL operation information) in the DDL log, determining whether a data table with changed table structure information exists in the database, and determining whether the data table is a source data table of the executing synchronization task, that is, by monitoring the ALTER record information in the DDL log, identifying the DDL operation condition, the monitoring of the table structure information of the source data table can be realized.
And 102, if the table structure information changes, modifying the task statement of the synchronous task according to the changed table structure information.
In step 102, if it is determined that the table structure information of the source data table related to the executing synchronization task is changed by monitoring the ALTER record information, and the change type is at least one of the first type change types, the task statement of the synchronization task needs to be modified according to the changed table structure information of the source data table, so as to avoid interruption of data synchronization caused by mismatching of the table structure information of the source data table and the task statement of the synchronization task.
For example, the currently executed synchronization task is to synchronize the field a1, the field a2, the field a3, the field a4, and the field a5 in the source data table a to the field B1, the field B2, the field B3, the field B4, and the field B5 corresponding to the destination data table B, and the task statement corresponding to the synchronization task may be represented as follows, but is not limited to:
truncate table TargetData.[dbo].[B.]
go
insert into TargetData.[dbo].[B]select*from SourceData.[dbo].[A]
B.column b1＝b.column a1；B.column b2＝b.column a2；B.column b3＝b.column a3；B.column b4＝b.column a4；B.column b5＝b.column a5
go
in one example, if the field a1 of the source data table is deleted during the execution of the synchronization task, the field a1 cannot be obtained at this time, so that the execution of the synchronization task is interrupted, and the synchronization operation of other data cannot be completed. The modified task statement may be represented, but is not limited to, as follows:
truncate table TargetData.[dbo].[B.]
go
insert into TargetData.[dbo].[B]select*from SourceData.[dbo].[A]
B.column b2＝b.column a2；B.column b3＝b.column a3；B.column b4＝b.column a4；B.column b5＝b.column a5
go
in another example, if the field a6 and its corresponding data are added to the source data table, the original synchronization task cannot synchronize the data corresponding to the newly added field to the destination data table B, and therefore the task statement of the synchronization task needs to be modified. The modified task statement may be represented, but is not limited to, as follows:
truncate table TargetData.[dbo].[B.]
go
insert into TargetData.[dbo].[B]select*from SourceData.[dbo].[A]
B.column b1＝b.column a1；B.column b2＝b.column a2；B.column b3＝b.column a3；B.column b4＝b.column a4；B.column b5＝b.column a5；B.column b6＝b.column a6
go
for the modification of the synchronization task, in one embodiment, the modification may be directly made on the task statement of the original synchronization task, so that the original synchronization task is directly replaced by the modified synchronization task.
For the modification of the synchronization task, in another embodiment, the original synchronization task may be copied first, and the task statement of the copied synchronization task is modified, so that not only is a new synchronization task created, but also the original synchronization task is retained, which is beneficial to tracing the history of the system running state, and certainly, the execution of the original task statement needs to be stopped.
And 103, executing the modified synchronization task, and continuing to implement data synchronization processing from the source data table to the destination data table.
In step 103, after the synchronization task is modified, the modified synchronization task is executed to synchronize the data of the incomplete synchronization operation in the source data table to the destination data table.
Therefore, the synchronous task is adaptively modified by monitoring the change of the table structure information of the source data table, the execution of the synchronous task is ensured not to be interrupted due to the change of the table structure information of the source data table, and the data synchronization operation is effectively completed.
In another embodiment, if the table structure information change of the source data table relates to the first type of change, it is sometimes necessary to update the table structure information of the destination data table according to the changed table structure information.
For example, in one example, the currently executed synchronization task is to synchronize the field a1, the field a2, the field a3, the field a4, and the field a5 in the source data table a to the field B1, the field B2, the field B3, the field B4, and the field B5 corresponding to the destination data table B, and the destination data table B sets all fields to be not allowed to be empty (Null). If the field a1 of the source data table is deleted, even after the synchronization task is modified, the source data table acquires data corresponding to 4 fields, and the data corresponding to the field a1 is lacked, so that the data acquired from the source data table cannot be written into the destination data table. Based on this, if the next data synchronization operation is to be completed, after the task statement of the synchronization task is modified, the table structure information of the destination data table needs to be modified, and the B1 field of the destination data table B is modified to be allowed to be Null.
In another example, the currently executed synchronization task is to synchronize the field a1, the field a2, the field a3, the field a4, and the field a5 in the source data table a to the corresponding field B1, the field B2, the field B3, the field B4, and the field B5 in the destination data table B. If the field a6 is added to the source data table a, it is impossible to determine which column in the data table B the data corresponding to the field a6 is written into in the data synchronization process because the field corresponding to the field a6 and the description information thereof are absent from the data table B. Based on this, to complete the next data synchronization, the table structure information of the destination data table needs to be modified, and a column of fields, for example, the field B6 is added to write the data of the field a6, and the field attribute description of the field B6 needs to be added according to the field attributes, such as the field type and the field length, of the field a6, in the destination data table B.
For updating the destination data table, in one embodiment, the originally existing destination data table in the destination database may be modified, so that the originally existing destination data table is replaced by the modified destination data table. And executing the modified synchronization task, and synchronizing the data acquired from the source data table into the modified destination data table.
For the update of the destination data table, in another embodiment, a new destination data table may also be created in the destination database according to the table structure information of the originally existing destination data table, and the table structure information of the newly created destination data table is modified according to the table structure change information of the source data table, so that not only the new destination data table but also the original destination data table is created in the destination database. And executing the modified synchronization task, and synchronizing the data obtained from the source data table into the newly created destination data table so as to be convenient for operation and maintenance personnel to distinguish which data are the data synchronized according to the task statements before modification and which data are the data synchronized according to the task statements after modification.
For example, if a sync task is used to synchronize the field Stu _ id (school number), the field name (name), the field sex (gender), and the field mark (achievement) in the source data table a (see table 1) to the field Stu _ id, the field name, the field sex, and the field mark in the destination data table B, when "2007070103 in table a; wangming; male; 552 "after the piece of data is synchronized into the destination data table B, the field address is added to the table a, and the source data table after the field is added is the table a' (see table 2). If a new destination data table is created in the destination database, two tables, table B and table B', will exist in the database. The target data table B obtained by executing the task statements of the synchronization task before modification is shown in table 3, and the target data table B' (newly created target data table) obtained by executing the task statements of the synchronization task after modification is shown in table 4.
TABLE 1
| Stu_id | name | sex | mark | sdept |
| 2007070101 | Zhang Yuan | For male | 555 | Information college |
| 2007070102 | Zhang hong | Woman | 565 | Information college |
| 2007070103 | Wangming liquor | For male | 552 | Information college |
| 2007070104 | Liwei | For male | 564 | Information college |
| 2007070201 | Zhenglan Lang | Woman | 567 | Computer school |
| 2007070202 | Zhao Heng | For male | 566 | Computer school |
| 2007070203 | Zhang lan | Woman | 571 | Computer school |
| 2007080101 | Liwei | For male | 578 | Computer school |
TABLE 2
TABLE 3
| Stu_id | name | sex | mark | sdept |
| 2007070101 | Zhang Yuan | For male | 555 | Information college |
| 2007070102 | Zhang hong | Woman | 565 | Information college |
| 2007070103 | Wangming liquor | For male | 552 | Information college |
TABLE 4
| Stu_id | name | sex | mark | sdept | address |
| 2007070104 | Liwei | For male | 564 | Information college | Hangzhou Zhejiang province |
| 2007070201 | Zhenglan Lang | Woman | 567 | Computer school | Zhejiang lake zhou |
| 2007070202 | Zhao Heng | For male | 566 | Computer school | Jiaxing Zhejiang |
| 2007070203 | Zhang lan | Woman | 571 | Computer school | Hangzhou Zhejiang province |
| 2007080101 | Liwei | For male | 578 | Computer school | Zhejiang Shaoxing wine |
For the case that the table structure information of the destination data table needs to be updated, in another embodiment, if the destination data table does not support the modification of the table structure information, or the modification of the table structure information fails, the data synchronization method may further generate alarm information to notify the operation and maintenance personnel to perform manual intervention. The alarm information may be, but is not limited to, notifying the operation and maintenance staff by means of short messages, mails, and the like.
In the data synchronization process, link tasks are involved, and the synchronization tasks with the association relationship are called link tasks. For a plurality of synchronous tasks in the link task, in the process of executing one synchronous task, the table structure information of the source data table related to the synchronous task changes, so that other tasks which are not executed and completed and are related to the source data table cannot be executed, and therefore before the other tasks which are not executed and completed are interrupted, all task statements of other tasks which are not executed and completed and are related to the source data table can be modified according to the changed table structure information of the source data table. Wherein, other tasks which are not executed and completed comprise: the synchronization task is used for implementing data synchronization operation of the source data table to other destination data tables; or, a synchronization task for performing a data synchronization operation of the destination data table to another destination data table.
For example, if there are three tasks to be executed sequentially, the three tasks are:
the synchronization task a is used for synchronizing the data corresponding to the field a1, the field a2, the field a3 and the field a4 in the data table A into the field B1, the field B2, the field B3 and the field B4 in the data table B;
the synchronization task B is used for synchronizing the data corresponding to the field B1, the field B2 and the field B3 in the data table B into the field C1, the field C2 and the field C3 in the data table C;
and the synchronization task C is used for synchronizing the data corresponding to the field C1, the field C2, the field C3 and the field C5 in the data table C into the field D1, the field D2, the field D3 and the field D5 in the data table D.
The data table B is a target data table of the synchronous task a and is a source data table of the synchronous task B; the data table C is a destination data table of the synchronous task b and is a source data table of the synchronous task C; the data tables related to the synchronous task a, the synchronous task b and the synchronous task c have an association relationship, and the synchronous task is called a link task.
If the field a1 in the data table a is deleted during the execution of the sync task a, not only the sync task a will be interrupted, but also the sync task b and the sync task c cannot be executed. Therefore, when the synchronous task a is modified or after the synchronous task a is modified, the synchronous task a and the synchronous task b need to be modified correspondingly according to the changed table structure information of the data table a.
The synchronization task can be used for performing data synchronization operation from one source data table to one destination data table, can also be used for performing data synchronization operation from a plurality of source data tables to one destination data table, and can also be used for performing data synchronization operation from one source data table to a plurality of destination data tables. When the synchronization task is used to synchronize target data in multiple source data tables to one destination data table, if the table structure information changes, including changes of associated fields of the multiple source data tables, such as deletion of the associated fields, or modification of names of the associated fields in one source data table, it is necessary to stop executing the synchronization task, and generate alarm information to notify operation and maintenance personnel to perform manual intervention. The alarm information may be, but is not limited to, notifying the operation and maintenance staff by means of short messages, mails, and the like.
For example, if the source database has tables 1 and 5, it is assumed that table 1 is a student information table, table 5 is a student household table, table 1 contains field Stu _ id, field name, field sex, field mark and field sdept (college), and table 5 contains field Stu _ id, field provice (birth province), field city (birth city) and field county (county), where field Stu _ id is the associated field of tables 1 and 5. If there is a synchronization task, the data corresponding to all fields in table 1 and table 5 need to be synchronized into a destination data table. In the process of executing the task, the field Stu _ id is deleted, since the field Stu _ id is a related field, in the process of synchronizing the data of the table 1 and the table 5 to the target data table, 2 data tables need to be spliced depending on the related field Stu _ id, and after the related field Stu _ id is deleted, the splicing of 2 tables cannot be performed, so that the synchronization of the data cannot be completed. Under the condition, the execution of the synchronous task is stopped, alarm information is generated and sent to operation and maintenance personnel for manual intervention.
TABLE 5
| Stu_id | province | city | county |
| 2007070101 | Zhejiang province | Hangzhou city | Bin river area |
| 2007070102 | Province of Hunan province | Hengyang city | Hengyang county |
| 2007070103 | Guangdong province | Huizhou city | Boruo county |
| 2007070104 | Zhejiang province | Hangzhou city | Region of the river |
| 2007070201 | Province of Hebei province | Handan market | Big county |
| 2007070202 | Province of Hebei province | City of Cangzhou | Qing county |
| 2007070203 | Shanxi province | Datong city | Guangling county |
| 2007080101 | Shanxi province | Changzhi city | Ride-on county |
Next, the data synchronization process of the embodiment of the present specification will be described in detail with respect to the second variation of the source data table.
FIG. 2 is a flow chart illustrating another method of data synchronization shown in the present specification according to an exemplary embodiment, including the steps of:
The synchronization task is used for implementing data synchronization operation from the source data table to the destination data table.
If the table structure information of the Data table in the database is modified, there is an ALTER (table structure modification) related record in a DDL (Data definitional language) log. Therefore, whether a data table with changed table structure information exists in the database can be confirmed by acquiring the ALTER record information in the DDL log, and whether the data table is a source data table of the executing synchronous task is determined, that is, the DDL operation condition can be identified by monitoring the ALTER record information in the DDL log, so that the monitoring of the table structure information of the source data table is realized.
In step 202, if it is determined that the table structure information of the source data table changes by monitoring the ALTER record information and the change type is at least one of the second changes, the table structure information of the destination data table needs to be updated according to the changed table structure information of the source data table, so as to avoid interruption of data synchronization caused by mismatch between the table structure information of the destination data table and the table structure information of the source data table.
For updating the destination data table, in one embodiment, the originally existing destination data table in the destination database may be modified, so that the originally existing destination data table is replaced by the modified destination data table. And executing the modified synchronization task, and synchronizing the data acquired from the source data table into the modified destination data table.
For the update of the destination data table, in another embodiment, a new destination data table may also be created in the destination database according to the table structure information of the originally existing destination data table, and the table structure information of the newly created destination data table is modified according to the table structure change information of the source data table, so that not only the new destination data table but also the original destination data table is created in the destination database. And executing the modified synchronization task, and synchronizing the data obtained from the source data table into the newly created destination data table so as to be convenient for operation and maintenance personnel to distinguish which data are the data synchronized according to the task statements before modification and which data are the data synchronized according to the task statements after modification.
And step 203, executing the task statement of the synchronization task, and continuing to implement the data synchronization processing from the source data table to the target data table.
If the table structure information of the data table is modified based on the original destination data table in step 202, the data acquired from the source data table is synchronized into the modified destination data table in the process of continuing the data synchronization process from the source data table to the destination data table in step 203.
If the destination data table is newly created in step 202, the data acquired from the source data table is synchronized into the newly created destination data table while the data synchronization process from the source data table to the destination data table is continued in step 203.
In another embodiment, if the destination data table does not support the modification of the table structure information, or the modification of the table structure information fails, step 203 is not executed, and an alarm message is generated to notify the operation and maintenance personnel to perform manual intervention. The alarm information may be, but is not limited to, notifying the operation and maintenance staff by means of short messages, mails, and the like.
In any of the above embodiments, a control policy may also be set, for example, the control policy a may be configured to generate alarm information when the table structure information of the source data table changes; the control strategy b is used for modifying the original target data table under the condition that the target data table needs to be updated; a control strategy c, under the condition that the target data table needs to be updated, newly building the target data table; according to the control strategy, a user can select one or more items.
Embodiments of an apparatus are also provided in the present specification, corresponding to embodiments of the method described above.
Fig. 3 is a block diagram of a data synchronization apparatus shown in the present specification according to an exemplary embodiment, the data synchronization apparatus including: a monitoring module 31, a modification module 32, and an execution module 33.
The monitoring module 31 is configured to monitor table structure information of a source data table in a process of executing a synchronization task, where the synchronization task is used to implement data synchronization operation from the source data table to a destination data table;
the modification module 32 is configured to modify the task statement of the synchronization task according to the changed table structure information when the table structure information changes;
the execution module 33 is configured to execute the modified synchronization task to continue to implement the data synchronization operation of the source data table to the destination data table.
An embodiment of the present invention further provides another data synchronization apparatus, where the data synchronization apparatus includes:
the system comprises a monitoring module, a synchronization module and a processing module, wherein the monitoring module is used for monitoring the table structure information of a source data table in the process of executing a synchronization task, and the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
the modification module is used for updating the table structure information of the target data table according to the changed table structure information under the condition that the table structure information is changed;
and the execution module is used for implementing the data synchronization operation of the source data table to the updated target data table.
Fig. 4 is a schematic diagram of an electronic device according to an exemplary embodiment of the present invention, showing a block diagram of an exemplary electronic device 40 suitable for use in implementing any of the embodiments of the present invention. The electronic device 40 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
As shown in FIG. 4, electronic device 40 may take the form of a general purpose computing device, which may be a server device, for example. The components of electronic device 40 may include, but are not limited to: the at least one processor 41, the at least one memory 42, and a bus 43 connecting the various system components (including the memory 42 and the processor 41).
The bus 43 includes a data bus, an address bus, and a control bus.
The memory 42 may include volatile memory, such as Random Access Memory (RAM)421 and/or cache memory 422, and may further include Read Only Memory (ROM) 423.
The processor 41 executes various functional applications and data processing, such as the methods provided by any of the above embodiments, by running a computer program stored in the memory 42.
The electronic device 40 may also communicate with one or more external devices 44 (e.g., keyboard, pointing device, etc.). Such communication may be through an input/output (I/O) interface 45. Also, the model-generated electronic device 40 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via a network adapter 46. As shown, the network adapter 46 communicates with the other modules of the model-generated electronic device 40 over a bus 43. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the model-generating electronic device 40, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of any one of the data synchronization methods described above.
The implementation process of the functions and actions of each module in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in the specification. One of ordinary skill in the art can understand and implement it without inventive effort.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Other embodiments of the present description will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the specification following, in general, the principles of the specification and including such departures from the present disclosure as come within known or customary practice within the art to which the specification pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the specification being indicated by the following claims.
It will be understood that the present description is not limited to the precise arrangements described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (12)
1. A data synchronization method, characterized in that the data synchronization method comprises:
monitoring table structure information of a source data table in the process of executing a synchronization task, wherein the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
if the table structure information changes, modifying the task statement of the synchronous task according to the changed table structure information;
and executing the modified synchronization task to continue to implement the data synchronization operation from the source data table to the destination data table.
2. The data synchronization method of claim 1, wherein if the table structure information changes, the data synchronization method further comprises:
updating the target data table according to the changed table structure information;
implementing a data synchronization operation of the source data table to the destination data table, including:
and implementing the data synchronization operation of the source data table to the updated destination data table.
3. The data synchronization method of claim 2, wherein updating the destination data table comprises:
correspondingly modifying the table structure information of the target data table according to the changed table structure information;
or, creating a new destination data table, wherein the table structure information of the new destination data table is the same as the changed table structure information.
4. The data synchronization method of claim 1, wherein if the table structure information changes, the data synchronization method further comprises:
and modifying all other tasks which are not executed and completed and are related to the source data table according to the changed table structure information.
5. The data synchronization method of claim 4, wherein the other unexecuted completed tasks include:
a synchronization task for implementing data synchronization operation of the source data table to other destination data tables;
or, a synchronization task for implementing data synchronization operation of the destination data table to other destination data tables.
6. The data synchronization method according to claim 1, wherein when the synchronization task is used to perform data synchronization operation from multiple source data tables to one destination data table, if the table structure information changes and includes a change in the associated fields of the multiple source data tables, the data synchronization method further comprises:
and stopping executing the synchronous task and generating alarm information.
7. The data synchronization method of claim 1, wherein the source data table is stored in a source database;
monitoring table structure information of a source data table, including:
acquiring a DDL log of the source database;
and monitoring the table structure information of the source data table according to the DDL operation information recorded in the DDL log.
8. A data synchronization method, characterized in that the data synchronization method comprises:
monitoring table structure information of a source data table in the process of executing a synchronization task, wherein the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
if the table structure information changes, updating the table structure information of the target data table according to the changed table structure information;
and implementing the data synchronization operation of the source data table to the updated destination data table.
9. A data synchronization apparatus, characterized in that the data synchronization apparatus comprises:
the system comprises a monitoring module, a synchronization module and a processing module, wherein the monitoring module is used for monitoring the table structure information of a source data table in the process of executing a synchronization task, and the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
the modification module is used for modifying the task statement of the synchronous task according to the changed table structure information under the condition that the table structure information is changed;
and the execution module is used for executing the modified synchronization task so as to continuously implement the data synchronization operation from the source data table to the destination data table.
10. A data synchronization apparatus, characterized in that the data synchronization apparatus comprises:
the system comprises a monitoring module, a synchronization module and a processing module, wherein the monitoring module is used for monitoring the table structure information of a source data table in the process of executing a synchronization task, and the synchronization task is used for implementing data synchronization operation from the source data table to a target data table;
the modification module is used for updating the table structure information of the target data table according to the changed table structure information under the condition that the table structure information is changed;
and the execution module is used for implementing the data synchronization operation of the source data table to the updated target data table.
11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data synchronization method of any one of claims 1 to 7 when executing the computer program.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data synchronization method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010121734.0A CN111367924B (en) | 2020-02-26 | 2020-02-26 | Data synchronization method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010121734.0A CN111367924B (en) | 2020-02-26 | 2020-02-26 | Data synchronization method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111367924A true CN111367924A (en) | 2020-07-03 |
| CN111367924B CN111367924B (en) | 2024-01-02 |
Family
ID=71206403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010121734.0A Active CN111367924B (en) | 2020-02-26 | 2020-02-26 | Data synchronization method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111367924B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112468697A (en) * | 2020-11-18 | 2021-03-09 | 江汉大学 | Visual acquisition method and device with intelligent position adjustment function |
| CN113204558A (en) * | 2021-05-25 | 2021-08-03 | 城云科技（中国）有限公司 | Method and device for automatically updating data table structure |
| WO2023029850A1 (en) * | 2021-09-01 | 2023-03-09 | 北京火山引擎科技有限公司 | Data processing method and apparatus, and electronic device and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
| CN108959564A (en) * | 2018-07-04 | 2018-12-07 | 玖富金科控股集团有限责任公司 | Data warehouse metadata management method, readable storage medium storing program for executing and computer equipment |
| CN109885581A (en) * | 2019-03-14 | 2019-06-14 | 苏州达家迎信息技术有限公司 | Synchronous method, device, equipment and the storage medium of database |
| CN110347693A (en) * | 2019-07-11 | 2019-10-18 | 招商局金融科技有限公司 | Method of data synchronization, device and storage medium |
| US20200019474A1 (en) * | 2016-11-29 | 2020-01-16 | Silcroad Soft, Inc. | Consistency recovery method for seamless database duplication |
-
2020
- 2020-02-26 CN CN202010121734.0A patent/CN111367924B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200019474A1 (en) * | 2016-11-29 | 2020-01-16 | Silcroad Soft, Inc. | Consistency recovery method for seamless database duplication |
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
| CN108959564A (en) * | 2018-07-04 | 2018-12-07 | 玖富金科控股集团有限责任公司 | Data warehouse metadata management method, readable storage medium storing program for executing and computer equipment |
| CN109885581A (en) * | 2019-03-14 | 2019-06-14 | 苏州达家迎信息技术有限公司 | Synchronous method, device, equipment and the storage medium of database |
| CN110347693A (en) * | 2019-07-11 | 2019-10-18 | 招商局金融科技有限公司 | Method of data synchronization, device and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112468697A (en) * | 2020-11-18 | 2021-03-09 | 江汉大学 | Visual acquisition method and device with intelligent position adjustment function |
| CN112468697B (en) * | 2020-11-18 | 2022-05-27 | 江汉大学 | Visual acquisition method and device with intelligent position adjustment function |
| CN113204558A (en) * | 2021-05-25 | 2021-08-03 | 城云科技（中国）有限公司 | Method and device for automatically updating data table structure |
| WO2023029850A1 (en) * | 2021-09-01 | 2023-03-09 | 北京火山引擎科技有限公司 | Data processing method and apparatus, and electronic device and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111367924B (en) | 2024-01-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104423960B (en) | A kind of method and system of project continuous integrating | |
| US10726042B2 (en) | Replication control using eventually consistent meta-data | |
| EP3477914B1 (en) | Data recovery method and device, and cloud storage system | |
| US9031910B2 (en) | System and method for maintaining a cluster setup | |
| CN111367924A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| US8805777B2 (en) | Data record collapse and split functionality | |
| US8285677B2 (en) | Method and apparatus for propagating tables while preserving cyclic foreign key relationships | |
| US11657111B2 (en) | Optimistic data fetching and rendering | |
| CN106933703A (en) | A kind of method of database data backup, device and electronic equipment | |
| US8417669B2 (en) | Auto-correction in database replication | |
| US20140289562A1 (en) | Controlling method, information processing apparatus, storage medium, and method of detecting failure | |
| US20200192888A1 (en) | Distributed trust data storage system | |
| US8386462B2 (en) | Standby index in physical data replication | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| CN114077518A (en) | Data snapshot method, device, equipment and storage medium | |
| CN115660620A (en) | Personnel DB library management method and device based on RPA and storage medium | |
| CN105550342B (en) | A kind of data processing method of the distributed data base of all-transparent | |
| CN105574026A (en) | Method and device for service supporting by using non-relational database | |
| CN114416868A (en) | Data synchronization method, device, equipment and storage medium | |
| US10873504B1 (en) | Managing concurrently received configuration requests in a computing network | |
| CN109710698A (en) | A kind of data assemblage method, device, electronic equipment and medium | |
| CN110727677A (en) | Method and device for tracing blood relationship of table in data warehouse | |
| CN111126961A (en) | Complex product full life cycle digital mainline service system | |
| CN109522098A (en) | Transaction methods, device, system and storage medium in distributed data base | |
| JP5832592B1 (en) | Data management device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |