CN110543513B - Incremental data synchronization method, equipment and storage medium for distributed database - Google Patents
Incremental data synchronization method, equipment and storage medium for distributed database Download PDFInfo
- Publication number
- CN110543513B CN110543513B CN201810533666.1A CN201810533666A CN110543513B CN 110543513 B CN110543513 B CN 110543513B CN 201810533666 A CN201810533666 A CN 201810533666A CN 110543513 B CN110543513 B CN 110543513B
- Authority
- CN
- China
- Prior art keywords
- data
- field
- distributed database
- change
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000008859 change Effects 0.000 claims abstract description 107
- 230000001360 synchronised effect Effects 0.000 claims abstract description 26
- 230000004048 modification Effects 0.000 claims description 16
- 238000012986 modification Methods 0.000 claims description 16
- 230000015654 memory Effects 0.000 claims description 14
- 238000001914 filtration Methods 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 abstract description 12
- 230000005540 biological transmission Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a distributed database incremental data synchronization method, which comprises the following steps: acquiring change data in the DBMS; converting the change data into a data change log; analyzing the data change log to generate a corresponding SQL statement; and synchronizing the change data to a distributed database through SQL sentences and according to distribution key information. In addition, the invention also discloses equipment and a storage medium. The invention can capture the change of DBMS data and synchronize the change to the distributed database in real time, and generate SQL synchronous data according to the distribution strategy, has powerful function and simple use, and is suitable for data transmission and synchronization of large data volume by using batch processing in the synchronization process.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a method, an apparatus, and a storage medium for synchronizing incremental data of a distributed database.
Background
In order to achieve real-time synchronization of data between a database management system (Database Management System, DBMS) (e.g., oracle, sybase and MS, SQL Server) and a distributed database, trigger capture is often employed when data synchronization is performed between existing heterogeneous databases. However, the trigger capture method is simple and clear, but has a large influence on the processing efficiency of the service, and is not suitable for use in a high-load service environment.
Disclosure of Invention
The invention mainly aims to provide a distributed database incremental data synchronization method, equipment and a storage medium, and aims to solve the problem that the existing heterogeneous database synchronization cannot be realized in a high-load service environment.
In order to achieve the above object, the present invention provides a distributed database incremental data synchronization method, which includes the steps of:
acquiring change data in a database management system DBMS;
converting the change data into a data change log;
analyzing the data change log to generate a corresponding Structured Query Language (SQL) statement;
and synchronizing the change data to the distributed database through the SQL statement according to the distribution information of the distributed database table.
In addition, in order to achieve the above object, the present invention also proposes an apparatus comprising a processor and a memory;
the processor is configured to execute a distributed database incremental data synchronization program stored in the memory to implement the method described above.
In addition, to achieve the above object, the present invention also proposes a computer-readable storage medium storing one or more programs executable by one or more processors to implement the above method.
The distributed database incremental data synchronization method, the distributed database incremental data synchronization device and the computer readable storage medium provided by the invention are capable of capturing and synchronizing the change of DBMS data to the distributed database in real time through the SQL statement and according to the distribution key information by acquiring the change data in the DBMS, converting the change data into the data change log and analyzing the data change log to generate the corresponding SQL statement, generating SQL synchronous data according to the distribution strategy, and are powerful in function and simple to use, and batch processing is used in the synchronization process to be suitable for data transmission and synchronization of a large amount of data.
Drawings
FIG. 1 is a flowchart of a method for incremental data synchronization of a distributed database according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram showing a sub-process of a method for synchronizing incremental data of a distributed database according to a first embodiment of the present invention;
FIG. 3 is a schematic diagram of another sub-flowchart of the incremental data synchronization method according to the first embodiment of the present invention;
FIG. 4 is a second schematic sub-flowchart of the incremental data synchronization method of the distributed database according to the first embodiment of the present invention;
FIG. 5 is a third schematic sub-flowchart of a method for synchronizing incremental data of a distributed database according to the first embodiment of the present invention;
FIG. 6 is another flow chart of the incremental data synchronization method of the distributed database according to the first embodiment of the present invention;
FIG. 7 is another flow chart of the incremental data synchronization method of the distributed database according to the first embodiment of the present invention;
fig. 8 is a schematic diagram of a device hardware architecture according to a second embodiment of the present invention;
FIG. 9 is a block diagram of the distributed database based incremental data synchronization process of FIG. 8.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
In the following description, suffixes such as "module", "component", or "unit" for representing elements are used only for facilitating the description of the present invention, and have no specific meaning per se. Thus, "module," "component," or "unit" may be used in combination.
First embodiment
Fig. 1 is a schematic flow chart of a distributed database incremental data synchronization method according to a first embodiment of the present invention. In fig. 1, the incremental data synchronization method of the distributed database includes:
step 110, change data in the DBMS is acquired.
In particular, change data is obtained for all data manipulation languages (Data Manipulation Language, DML) on a database management system (Database Management System, DBMS).
Step 120, converting the change data into a data change log.
Specifically, the operation types of the data change log include INSERT, UPDATE, and DELETE.
And 130, analyzing the data change log to generate a corresponding SQL statement.
Specifically, the data change log of the user is analyzed, wherein the data change log comprises operation types, data before and after the change and the like. Based on the parsing result, a partially structured query language (Structured Query Language, SQL) is spelled out.
And 140, synchronizing the change data to the distributed database through the SQL statement according to the distribution information of the distributed database table.
Specifically, the data is synchronized to the database, which may be a distributed database, by the generated SQL statement and according to the distribution key information.
Alternatively, in this embodiment, a registry is used to record the names of tables (simply referred to as table names) of all the data change logs to be captured, and the tables are referred to as a synchronization source table, in which information such as the table names of the changes, the user names to which the tables belong, the operation types, and the like are recorded in detail. After capturing the data change log, analyzing and restoring the data of the data change log to obtain a DML statement. Different operation types and different modes of restoring the DML statement, and SQL statements needing to be executed can be restored according to the operation types, table names and modified field values for the INSERT statement; for UPDATE and DELETE statements, because of the presence of a sphere condition in the statement, if all fields are spelled one by one into a sphere clause, not only does the statement be made long, for example: some tables of the service may have hundreds of fields, and due to the limitation of the distributed database, indexes cannot be used when the distributed database is executed, so that the characteristics of the distributed database are considered, and synchronization of UPDATE and DELETE sentences is optimized based on the distribution attribute and the distribution node information of the distributed database.
According to the method, the device and the system, the change of DBMS data is captured, the change data log file is analyzed, the SQL statement is generated, the data are synchronized according to the distribution strategy based on the characteristics of the distributed database, the data synchronization speed is improved, meanwhile, the distributed database is used for batch processing in the synchronization process, the performance is improved, the correctness of the data is checked through the script after the data synchronization is completed, and the consistency of the data is guaranteed.
Optionally, as shown in fig. 2, if the operation type of the parsed data change log is INSERT, step 140 specifically includes:
step 210, obtaining the names and field modification values of the tables stored in the data change log;
step 220, according to the field modification value, obtaining a field name list and a field value list;
step 230, selecting a field value from the field name list and the field value list;
step 240, determining whether the field value is NULL; if yes, go to step 250, if not, go to step 260;
step 250, not synchronizing the field values;
step 260, synchronizing the field values.
Specifically, the names of the tables are obtained according to the object_name stored in the data change log, the field name list and the field value list are obtained according to the field modification value (new_value_list) stored in the data change log, when the field value is judged to be NULL, the fact that the source sentence is not assigned is indicated, the field should not appear in the field list of the insert sentence, the field value does not need to be synchronized, and otherwise the field value needs to be synchronized.
Optionally, as shown in fig. 3, after step 250 or step 260, the incremental data synchronization method of the distributed database of this embodiment further includes:
step 310, determining whether the field value is the last field in the field name list and the field value list; if yes, go to step 320, if no, go to step 330;
step 320, issuing the field value to a preset storage node;
and step 330, re-selecting another field value from the field name list and the field value list.
Specifically, after determining that the field value is NULL and synchronization is not required, and determining that the field value is not NULL and synchronization is required, it is further required to determine whether the field value is the last field in the field name list and the value list, if so, then the field value is issued to a storage node of the preset database, otherwise, another field value is selected from the field name list and the value list, and the process of steps 210-260 is repeated until the determination of all the field values is completed.
Optionally, as shown in fig. 4, if the operation type of the parsed data change log is UPDATE, step 140 specifically includes:
step 410, obtaining the names and field modification values of the tables stored in the data change log;
step 420, obtaining the field name and the field value appearing in the field modification value to spell out the clause of the preset type;
step 430, querying distribution key information of the table;
step 440, filtering out the distribution key field of the table in the database in the field value and converting the distribution key field into a preset condition;
step 450, synchronizing the field values to the database.
Specifically, the name of the table is obtained according to the object_name stored in the data change log, and a clause of a preset type is spelled out according to the field name and the field value appearing in the field modification value (new_value_list) stored in the data change log, wherein the clause of the preset type can be a set clause. And inquiring distribution key information of a table from the data cache, and filtering out distribution key fields of the table in the distributed database from field values to splice preset conditions, wherein the preset conditions can be where conditions. And synchronizing to middleware of the distributed database according to the granularity of SQL sentences, so that the middleware can issue sentences needing to be synchronized to a designated database storage node according to the distribution attribute.
Optionally, as shown in fig. 5, if the operation type of the parsed data change log is DELETE, step 140 specifically includes:
step 510, obtaining names of tables stored in the data change log;
step 520, querying distribution key information of the table;
step 530, filtering out the distribution key field of the table in the database in the field value and converting the distribution key field into a preset condition;
step 540, synchronizing the field values to the database.
Specifically, the name of the table is obtained according to the object_name stored in the data change log, the distribution key information of the table is queried from the data cache, and the distribution key field of the table in the distributed database is filtered out from the field value, so that a preset condition is spliced, wherein the preset condition can be a sphere condition. And synchronizing to middleware of the distributed database according to the granularity of SQL sentences, so that the middleware can issue sentences needing to be synchronized to a designated database storage node according to the distribution attribute.
Optionally, as shown in fig. 6, if the operation type of the parsed data change log is not INSERT, UPDATE, DELETE, the incremental data synchronization method of the distributed database of the embodiment further includes:
step 610, determining whether the change data is synchronized to the database; if not, go to step 620;
step 620, uploading an alarm;
step 630, classifying the reported alarm types;
step 640, determining whether the alarm is a severe alarm; if yes, go to step 650;
in step 650, an alert prompt is generated.
Specifically, when the incremental data synchronization of the distributed database fails, alarm processing is performed, the uploaded alarms are checked, and alarm types are classified, and in this embodiment, the alarm types are classified into general alarms and serious alarms. If the alarm is a general alarm, the alarm is not processed, and if the alarm is a serious alarm, an alarm prompt is generated and manual intervention is carried out.
Optionally, as shown in fig. 7, after step 140, the incremental data synchronization method of the distributed database of this embodiment further includes:
step 710, perfecting the SQL statement according to the distribution key information and the distribution attribute of the table involved in the SQL statement;
step 720, sending data to a preset data storage node through the completed SQL sentence, so that the data storage node executes the completed SQL sentence and obtains an execution result;
and step 730, receiving and checking the execution result.
Specifically, after the change data is synchronized to the database, the SQL statement is perfected according to the distribution key information and the distribution attribute of the table related in the SQL statement, the data is distributed to the storage node of the appointed database according to the distribution characteristic, so that the storage node executes the SQL statement, the execution result of the SQL statement by the storage node is received, and the data synchronization consistency check is carried out on the execution result.
According to the incremental data synchronization method for the distributed database, the change data in the DBMS is obtained, the change data are converted into the data change log, the data change log is analyzed to generate corresponding SQL sentences, the change data are synchronized to the distributed database through the SQL sentences according to the distribution key information, the change of the DBMS data can be captured and synchronized to the distributed database in real time, SQL synchronous data are generated according to the distribution strategy, the functions are powerful and the use is simple, and batch processing is used in the synchronization process to be suitable for data transmission and synchronization of large data quantity.
Second embodiment
As shown in fig. 8, a schematic diagram of a device hardware architecture is provided in a second embodiment of the present invention. In fig. 8, the apparatus includes: memory 810, processor 820, and distributed database delta data synchronization program 830 stored on the memory 810 and executable on the processor 820. In this embodiment, the distributed database incremental data synchronization program 830 includes a series of computer program instructions stored on the memory 810 that, when executed by the processor 820, perform the distributed database incremental data synchronization operations of various embodiments of the present invention. In some embodiments, the distributed database delta data synchronization program 830 may be divided into one or more modules based on the particular operations implemented by portions of the computer program instructions. As shown in fig. 9, the distributed database incremental data synchronization program 830 includes: an acquisition module 910, a change data conversion module 920, an analysis module 930, a synchronization module 940, a judgment module 950, an alarm processing module 960, a transmission module 970, and a verification module 980. Wherein,
an acquisition module 910, configured to acquire the change data in the DBMS.
Specifically, the acquisition module 910 acquires the change data of all DMLs on the DBMS.
The change data conversion module 920 is configured to convert the change data into a data change log.
Specifically, the operation types of the data change log include INSERT, UPDATE, and DELETE.
And the parsing module 930 is configured to parse the data change log to generate a corresponding SQL statement.
Specifically, the parsing module 930 parses the user's data change log, including the operation type, the data before and after the change, and the like. And according to the analysis result, spelling out part of SQL.
And the synchronization module 940 is configured to synchronize the change data to the distributed database according to the distribution information of the distributed database table by using the SQL statement.
Specifically, the synchronization module 940 synchronizes data to a database, which may be a distributed database, through the generated SQL statement and according to the distribution key information.
Alternatively, in this embodiment, a registry is used to record the names of tables (simply referred to as table names) of all the data change logs to be captured, and the tables are referred to as a synchronization source table, in which information such as the table names of the changes, the user names to which the tables belong, the operation types, and the like are recorded in detail. After capturing the data change log, analyzing and restoring the data of the data change log to obtain a DML statement. Different operation types and different modes of restoring the DML statement, and SQL statements needing to be executed can be restored according to the operation types, table names and modified field values for the INSERT statement; for UPDATE and DELETE statements, because of the presence of a sphere condition in the statement, if all fields are spelled one by one into a sphere clause, not only does the statement be made long, for example: some tables of the service may have hundreds of fields, and due to the limitation of the distributed database, indexes cannot be used when the distributed database is executed, so that the characteristics of the distributed database are considered, and synchronization of UPDATE and DELETE sentences is optimized based on the distribution attribute and the distribution node information of the distributed database.
According to the method, the device and the system, the change of DBMS data is captured, the change data log file is analyzed, the SQL statement is generated, the data are synchronized according to the distribution strategy based on the characteristics of the distributed database, the data synchronization speed is improved, meanwhile, the distributed database is used for batch processing in the synchronization process, the performance is improved, the correctness of the data is checked through the script after the data synchronization is completed, and the consistency of the data is guaranteed.
If the operation type of the parsed data change log is INSERT, the synchronization module 940 is specifically configured to:
acquiring names and field modification values of tables stored in the data change log;
acquiring a field name list and a field value list according to the field modification value;
selecting a field value from the field name list and the field value list;
judging whether the field value is NULL; if yes, the field values are not synchronized, and if not, the field values are synchronized.
Specifically, the names of the tables are obtained according to the object_name stored in the data change log, the field name list and the field value list are obtained according to the field modification value (new_value_list) stored in the data change log, when the field value is judged to be NULL, the fact that the source sentence is not assigned is indicated, the field should not appear in the field list of the insert sentence, the field value does not need to be synchronized, and otherwise the field value needs to be synchronized.
Optionally, the synchronization module 940 is further configured to: judging whether the field value is the last field in the field name list and the field value list; if yes, the field value is issued to a preset storage node, and if not, the other field value is selected from the field name list and the field value list again.
Specifically, after determining that the field value is NULL and synchronization is not required, and determining that the field value is not NULL and synchronization is required, it is further required to determine whether the field value is the last field in the field name list and the value list, if so, then the field value is issued to a storage node of the preset database, otherwise, another field value is selected from the field name list and the value list, and the process of steps 210-260 is repeated until the determination of all the field values is completed.
Optionally, if the operation type of the parsed data change log is UPDATE, the synchronization module 940 is specifically configured to:
acquiring names and field modification values of tables stored in the data change log;
acquiring a field name and a field value appearing in the field modification value to spell clauses of a preset type;
querying distribution key information of the table;
filtering distribution key fields of the table in the database from the field values and converting the distribution key fields into preset conditions;
synchronizing the field values to the database.
Specifically, the name of the table is obtained according to the object_name stored in the data change log, and a clause of a preset type is spelled out according to the field name and the field value appearing in the field modification value (new_value_list) stored in the data change log, wherein the clause of the preset type can be a set clause. And inquiring distribution key information of a table from the data cache, and filtering out distribution key fields of the table in the distributed database from field values to splice preset conditions, wherein the preset conditions can be where conditions. And synchronizing to middleware of the distributed database according to the granularity of SQL sentences, so that the middleware can issue sentences needing to be synchronized to a designated database storage node according to the distribution attribute.
Optionally, if the operation type of the parsed data change log is DELETE, the synchronization module 940 is specifically configured to:
acquiring names of tables stored in the data change log;
querying distribution key information of the table;
filtering distribution key fields of the table in the database from the field values and converting the distribution key fields into preset conditions;
synchronizing the field values to the database.
Specifically, the name of the table is obtained according to the object_name stored in the data change log, the distribution key information of the table is queried from the data cache, and the distribution key field of the table in the distributed database is filtered out from the field value, so that a preset condition is spliced, wherein the preset condition can be a sphere condition. And synchronizing to middleware of the distributed database according to the granularity of SQL sentences, so that the middleware can issue sentences needing to be synchronized to a designated database storage node according to the distribution attribute.
A judging module 950, configured to judge whether the change data is synchronized to the database when the operation type of the parsed data change log is not INSERT, UPDATE, DELETE.
The alarm processing module 960 is configured to upload an alarm and classify the type of the reported alarm when the judging module 950 judges that the synchronization of the change data fails;
the judging module 950 is further configured to judge whether the alarm is a serious alarm; if so, the alarm processing module 960 is also configured to generate an alarm prompt.
Specifically, when the incremental data synchronization of the distributed database fails, the alarm processing module 960 performs alarm processing, checks the uploaded alarms and classifies the alarm types, in this embodiment, the alarm types are classified into general alarms and serious alarms. If the alarm is a general alarm, the alarm is not processed, and if the alarm is a serious alarm, an alarm prompt is generated and manual intervention is carried out.
A sending module 970, configured to perfect the SQL statement according to the distribution key information and the distribution attribute of the table related in the SQL statement, and send data to a preset data storage node through the perfect SQL statement, so that the data storage node executes the perfect SQL statement and obtains an execution result;
and the verification module 980 is configured to receive and verify the execution result.
Specifically, after the change data is synchronized to the database, the SQL statement is perfected according to the distribution key information and the distribution attribute of the table involved in the SQL statement, and the sending module 970 distributes the data to the storage node of the designated database according to the distribution characteristic, so that the storage node executes the SQL statement, and the checking module 980 receives the execution result of the SQL statement by the storage node and performs data synchronization consistency check on the execution result.
According to the device of the embodiment, the change data in the DBMS is acquired through the acquisition module 910, the change data conversion module 920 converts the change data into a data change log, the analysis module 930 analyzes the data change log to generate a corresponding SQL statement, the synchronization module 940 synchronizes the change data to the distributed database through the SQL statement according to the distribution key information, the change of the DBMS data can be captured and synchronized to the distributed database in real time, SQL synchronous data is generated according to the distribution strategy, the function is powerful and the use is simple, and batch processing is used in the synchronization process to be suitable for data transmission and synchronization of large data volume.
Third embodiment
The embodiment of the invention also provides a computer readable storage medium. The computer-readable storage medium here stores one or more programs. Wherein the computer readable storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid state disk; the memory may also comprise a combination of the above types of memories. The one or more programs in the computer-readable storage medium may be executed by one or more processors to implement the incremental data synchronization sub-method for a distributed database provided in the first embodiment described above.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (9)
1. A method for incremental data synchronization for a distributed database, the method comprising the steps of:
acquiring change data in a database management system DBMS;
converting the change data into a data change log;
analyzing the data change log to generate a corresponding Structured Query Language (SQL) statement;
synchronizing the change data to a distributed database according to the SQL statement and the distribution information of the distributed database table;
wherein after synchronizing the change data to the distributed database, the method further comprises:
perfecting the SQL statement according to distribution key information and distribution attribute of a table related in the SQL statement;
transmitting data to a preset data storage node through the completed SQL sentence, so that the data storage node executes the completed SQL sentence and obtains an execution result;
and receiving and checking the execution result.
2. The method of claim 1, wherein synchronizing the change data to the distributed database if the parsed operation type of the data change log is INSERT comprises:
acquiring names and field modification values of tables stored in the data change log;
acquiring a field name list and a field value list according to the field modification value;
selecting a field value from the field name list and the field value list;
judging whether the field value is NULL;
if yes, the field values are not synchronized;
if not, synchronizing the field value.
3. The distributed database incremental data synchronization method of claim 2 wherein after determining not to synchronize the field values or to synchronize the field values, the method further comprises:
judging whether the field value is the last field in the field name list and the field value list;
if yes, the field value is issued to a preset storage node;
if not, re-selecting another field value from the field name list and the field value list.
4. The incremental data synchronization method of claim 1 wherein synchronizing the change data to the distributed database if the parsed operation type of the data change log is UPDATE comprises:
acquiring names and field modification values of tables stored in the data change log;
acquiring a field name and a field value appearing in the field modification value to spell clauses of a preset type;
querying distribution key information of the table;
filtering distribution key fields of the table in the database from the field values and converting the distribution key fields into preset conditions;
synchronizing the field values to the database.
5. The method of claim 1, wherein synchronizing the change data to the distributed database if the parsed operation type of the data change log is DELETE comprises:
acquiring names of tables stored in the data change log;
querying distribution key information of the table;
filtering distribution key fields of the table in the database from field values and converting the distribution key fields into preset conditions;
synchronizing the field values to the database.
6. The method of claim 1, further comprising, if the parsed operation type of the data change log is not INSERT, UPDATE, DELETE:
judging whether the change data is synchronous to the database;
if not, uploading the alarm.
7. The distributed database incremental data synchronization method of claim 6 wherein after uploading the alert, the method further comprises:
classifying the reported alarm types;
judging whether the alarm is a serious alarm or not;
if yes, an alarm prompt is generated.
8. An apparatus comprising a processor and a memory;
the processor is configured to execute a distributed database incremental data synchronization program stored in the memory to implement the method of any one of claims 1-7.
9. A computer readable storage medium storing one or more programs executable by one or more processors to implement the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533666.1A CN110543513B (en) | 2018-05-29 | 2018-05-29 | Incremental data synchronization method, equipment and storage medium for distributed database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533666.1A CN110543513B (en) | 2018-05-29 | 2018-05-29 | Incremental data synchronization method, equipment and storage medium for distributed database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110543513A CN110543513A (en) | 2019-12-06 |
| CN110543513B true CN110543513B (en) | 2024-03-15 |
Family
ID=68701585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810533666.1A Active CN110543513B (en) | 2018-05-29 | 2018-05-29 | Incremental data synchronization method, equipment and storage medium for distributed database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110543513B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143470A (en) * | 2019-12-27 | 2020-05-12 | 中国银行股份有限公司 | Cross-platform database data synchronization comparison method and device |
| CN111427898A (en) * | 2020-03-16 | 2020-07-17 | 上海爱数信息技术股份有限公司 | Continuous data protection system and method based on analysis of Oracle log |
| CN111597257A (en) * | 2020-05-09 | 2020-08-28 | 远光软件股份有限公司 | Database synchronization method and device, storage medium and terminal |
| CN112231396A (en) * | 2020-09-15 | 2021-01-15 | 山东浪潮通软信息科技有限公司 | Method for incrementally synchronizing data across databases |
| CN112395360B (en) * | 2020-12-01 | 2023-06-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN113792066A (en) * | 2021-09-23 | 2021-12-14 | 中国农业银行股份有限公司 | Dynamic analysis scheduling method and device for batch SQL |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187888A (en) * | 2007-12-11 | 2008-05-28 | 浪潮电子信息产业股份有限公司 | Method for coping database data in heterogeneous environment |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| EP2891994A1 (en) * | 2013-11-04 | 2015-07-08 | Guangdong Electronics Industry Institute Ltd. | Method for achieving automatic synchronization of multisource heterogeneous data resources |
| CN105005618A (en) * | 2015-07-21 | 2015-10-28 | 杭州合众数据技术有限公司 | Data synchronization method and system among heterogeneous databases |
| CN105447054A (en) * | 2014-09-25 | 2016-03-30 | 中兴通讯股份有限公司 | Data synchronization method, device and system for database |
| CN105824865A (en) * | 2015-12-09 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed type database data synchronization system |
| CN106250543A (en) * | 2016-08-10 | 2016-12-21 | 深圳市彬讯科技有限公司 | A kind of automation data inquiry synchronous storage method |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| KR101825910B1 (en) * | 2016-08-31 | 2018-03-22 | 국방과학연구소 | Heterogeneous database synchronization system and method using neutral data based on redo log analysis |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9607065B2 (en) * | 2013-04-26 | 2017-03-28 | Netapp, Inc. | Hierarchical coherency log for managing a distributed data storage system |
| CN103617176B (en) * | 2013-11-04 | 2017-03-15 | 广东电子工业研究院有限公司 | One kind realizes the autosynchronous method of multi-source heterogeneous data resource |
| US10671642B2 (en) * | 2016-11-11 | 2020-06-02 | International Business Machines Corporation | Copying data changes to a target database |
-
2018
- 2018-05-29 CN CN201810533666.1A patent/CN110543513B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187888A (en) * | 2007-12-11 | 2008-05-28 | 浪潮电子信息产业股份有限公司 | Method for coping database data in heterogeneous environment |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| EP2891994A1 (en) * | 2013-11-04 | 2015-07-08 | Guangdong Electronics Industry Institute Ltd. | Method for achieving automatic synchronization of multisource heterogeneous data resources |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| CN105447054A (en) * | 2014-09-25 | 2016-03-30 | 中兴通讯股份有限公司 | Data synchronization method, device and system for database |
| CN105005618A (en) * | 2015-07-21 | 2015-10-28 | 杭州合众数据技术有限公司 | Data synchronization method and system among heterogeneous databases |
| CN105824865A (en) * | 2015-12-09 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed type database data synchronization system |
| CN106250543A (en) * | 2016-08-10 | 2016-12-21 | 深圳市彬讯科技有限公司 | A kind of automation data inquiry synchronous storage method |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| KR101825910B1 (en) * | 2016-08-31 | 2018-03-22 | 국방과학연구소 | Heterogeneous database synchronization system and method using neutral data based on redo log analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110543513A (en) | 2019-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110543513B (en) | Incremental data synchronization method, equipment and storage medium for distributed database | |
| US11157512B2 (en) | Method and system for replicating data to heterogeneous database and detecting synchronization error of heterogeneous database through SQL packet analysis | |
| CN109656934B (en) | Source Oracle database DDL synchronization method and device based on log analysis | |
| CN108509658B (en) | XML file parsing method and device | |
| CN110569214B (en) | Index construction method and device for log file and electronic equipment | |
| CN109739867B (en) | Industrial metadata management method and system | |
| CN110263009B (en) | Method, device and equipment for generating log classification rule and readable storage medium | |
| US20060161558A1 (en) | Schema conformance for database servers | |
| CN108121827B (en) | Full data synchronization method and device | |
| CN111857880B (en) | Dialogue configuration item information management method, device, equipment and storage medium | |
| CN108920607B (en) | Method and device for discovering field and electronic equipment | |
| CN106209431A (en) | A kind of Approaches of Alarm Correlation and network management system | |
| CN111190807A (en) | Buried point testing method and device | |
| CN111625540A (en) | Method and device for verifying data synchronization integrity of relational database | |
| CN112257032A (en) | Method and system for determining APP responsibility subject | |
| CN111930842B (en) | Data checking method and device | |
| CN105719072B (en) | System and method for associating multi-segment component transactions | |
| CN112612802A (en) | Real-time data middlebox processing method, device and platform | |
| CN114911872A (en) | Intranet and extranet data synchronization method, device and system, extranet server and storage medium | |
| CN110248030B (en) | Method, device, equipment and storage medium for testing short message compensation mechanism | |
| CN111045917A (en) | Method and device for converting format of test case | |
| CN112597207B (en) | Metadata management system | |
| CN116798650A (en) | Epidemiological close contact person searching method based on big data | |
| CN116225868A (en) | Data acquisition processing method, device, equipment and medium of database | |
| CN117874033A (en) | Database sub-table method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220112 Address after: 100176 602, floor 6, building 6, courtyard 10, KEGU 1st Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang group, high-end industrial area of Beijing Pilot Free Trade Zone) Applicant after: Jinzhuan Xinke Co.,Ltd. Address before: 518000 Zhongnan communication tower, South China Road, Nanshan District high tech Industrial Park, Shenzhen, Guangdong Applicant before: ZTE Corp. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |