CN110543513A - incremental data synchronization method, equipment and storage medium for distributed database - Google Patents
incremental data synchronization method, equipment and storage medium for distributed database Download PDFInfo
- Publication number
- CN110543513A CN110543513A CN201810533666.1A CN201810533666A CN110543513A CN 110543513 A CN110543513 A CN 110543513A CN 201810533666 A CN201810533666 A CN 201810533666A CN 110543513 A CN110543513 A CN 110543513A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- distributed database
- change
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000008859 change Effects 0.000 claims abstract description 107
- 230000001360 synchronised effect Effects 0.000 claims abstract description 27
- 230000004048 modification Effects 0.000 claims description 17
- 238000012986 modification Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 14
- 238000001914 filtration Methods 0.000 claims description 8
- 238000013500 data storage Methods 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 9
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for synchronizing incremental data of a distributed database, which comprises the following steps: acquiring change data in the DBMS; converting the change data into a data change log; analyzing the data change log to generate a corresponding SQL statement; and synchronizing the change data to the distributed database through SQL statements and according to the distribution key information. In addition, the invention also discloses equipment and a storage medium. The method can capture the change of the DBMS data and synchronize the DBMS data to the distributed database in real time, generates the SQL synchronous data according to the distribution strategy, has strong function and simple use, and uses batch processing in the synchronization process to be suitable for data transmission and synchronization of large data volume.
Description
Technical Field
the invention relates to the technical field of databases, in particular to a method, equipment and a storage medium for incremental data synchronization of a distributed database.
background
In order to implement real-time data synchronization between a Database Management System (DBMS) (e.g., oracle, Sybase and MS, SQL Server) and a distributed Database, triggers are often used to capture data when data synchronization is performed between existing heterogeneous databases. However, although the trigger capture method is simple and clear, it will greatly affect the processing efficiency of the service, and is not suitable for being used in a high-load service environment.
Disclosure of Invention
The invention mainly aims to provide a method, equipment and a storage medium for incremental data synchronization of a distributed database, and aims to solve the problem that the existing heterogeneous database synchronization cannot be realized in a high-load service environment.
in order to achieve the above object, the present invention provides a method for incremental data synchronization of a distributed database, comprising the steps of:
Acquiring change data in a database management system (DBMS);
converting the change data into a data change log;
analyzing the data change log to generate a corresponding Structured Query Language (SQL) statement;
And synchronizing the change data to the distributed database through the SQL statement according to the distribution information of the distributed database table.
In addition, to achieve the above object, the present invention also provides an apparatus, which includes a processor and a memory;
the processor is configured to execute a distributed database incremental data synchronization program stored in the memory to implement the method described above.
further, to achieve the above object, the present invention also proposes a computer-readable storage medium storing one or more programs, which are executable by one or more processors to implement the above-described method.
the incremental data synchronization method, the incremental data synchronization equipment and the computer-readable storage medium for the distributed database provided by the invention have the advantages that the change data in the DBMS are obtained, the change data are converted into the data change log, the data change log is analyzed to generate the corresponding SQL statement, the change data are synchronized to the distributed database through the SQL statement and according to the distribution key information, the change of the DBMS data can be captured and synchronized to the distributed database in real time, the SQL synchronous data are generated according to the distribution strategy, the function is strong, the use is simple, and batch processing is used in the synchronization process to be suitable for data transmission and synchronization of large data volume.
drawings
Fig. 1 is a schematic flowchart of a distributed database incremental data synchronization method according to a first embodiment of the present invention;
Fig. 2 is a first sub-flow diagram illustrating a distributed database incremental data synchronization method according to a first embodiment of the present invention;
fig. 3 is a schematic view illustrating another sub-flow of a method for incremental data synchronization of a distributed database according to a first embodiment of the present invention;
Fig. 4 is a schematic sub-flow diagram of a distributed database incremental data synchronization method according to a first embodiment of the present invention;
Fig. 5 is a schematic sub-flow diagram of a distributed database incremental data synchronization method according to a first embodiment of the present invention;
Fig. 6 is another schematic flowchart of a distributed database incremental data synchronization method according to the first embodiment of the present invention;
Fig. 7 is another schematic flowchart of a distributed database incremental data synchronization method according to the first embodiment of the present invention;
FIG. 8 is a diagram illustrating a hardware architecture of a device according to a second embodiment of the present invention;
fig. 9 is a block diagram illustrating the incremental data synchronization procedure based on the distributed database in fig. 8.
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.
first embodiment
fig. 1 is a schematic flow chart of a distributed database incremental data synchronization method according to a first embodiment of the present invention. In fig. 1, the incremental data synchronization method for distributed databases includes:
At step 110, change data in the DBMS is obtained.
specifically, change Data of all Data Manipulation Languages (DMLs) on a Database Management System (DBMS) is acquired.
and step 120, converting the change data into a data change log.
Specifically, the operation types of the data change log include INSERT INSERT, UPDATE UPDATE, and DELETE DELETE.
step 130, analyzing the data change log to generate a corresponding SQL statement.
Specifically, a data change log of the user is analyzed, wherein the data change log comprises operation types, data before and after change and the like. Based on the parsing result, a partially Structured Query Language (SQL) is spelled out.
And step 140, synchronizing the change data to the distributed database through the SQL statement according to the distribution information of the distributed database table.
specifically, the database may be a distributed database by synchronizing data to the database according to the distribution key information by the generated SQL statement.
optionally, in this embodiment, a registry is used to record names (abbreviated as table names) of all tables of the data change log to be captured, the table is called a synchronization source table, and information such as a table name of a change, a user name to which the table belongs, and an operation type is recorded in detail in the synchronization source table. After the data change log is captured, the DML statement is restored through data analysis of the data change log. Different operation types and different modes for restoring the DML statement are adopted, and for the INSERT statement, the SQL statement to be executed can be restored according to the operation type, the table name and the modified field value; for UPDATE and DELETE statements, with the where condition in the statement, if all fields are pieced together one by one into the where clause, it not only makes the statement very long, for example: some tables of the service may have hundreds of fields, and due to the limitation of the distributed database, the index cannot be used during the execution on the distributed database side, so that the synchronization of the UPDATE and DELETE statements is optimized based on the distribution attribute and the distribution node information of the distributed database in consideration of the characteristics of the distributed database.
In the embodiment, the change of the DBMS data is captured, the change data log file is analyzed, the SQL statement is generated, the data is synchronized according to the distribution strategy based on the characteristics of the distributed database, the data synchronization speed is improved, meanwhile, the distributed database is used for batch processing in the synchronization process, the performance is improved, the data correctness is verified through the script after the data synchronization is completed, and the data consistency is ensured.
optionally, as shown in fig. 2, if the operation type of the analyzed data change log is INSERT, step 140 specifically includes:
step 210, obtaining the name and the field modification value of the table stored in the data change log;
Step 220, acquiring a field name list and a field value list according to the field modification value;
Step 230, selecting a field value from the field name list and the field value list;
Step 240, judging whether the field value is NULL; if yes, go to step 250, otherwise go to step 260;
Step 250, not synchronizing said field values;
step 260, synchronizing the field values.
specifically, the name of the table is obtained according to the object _ name stored in the data change log, and the field name list and the field value list are obtained according to the field modification value (new _ value _ list) stored in the data change log, when the field value is judged to be NULL, it is indicated that the source statement is not assigned, the field should not appear in the field list of the insert statement, the field value does not need to be synchronized, and otherwise, the field value needs to be synchronized.
optionally, as shown in fig. 3, after step 250 or step 260, the incremental data synchronization method for a distributed database of this embodiment further includes:
step 310, judging whether the field value is the last field in the field name list and the field value list; if yes, go to step 320, otherwise go to step 330;
Step 320, sending the field value to a preset storage node;
in step 330, another field value is selected from the field name list and the field value list again.
Specifically, after the field value is judged to be NULL and not to be synchronized, and the field value is judged not to be NULL and to be synchronized, whether the field value is the last field in the field name list and the value list needs to be judged, if yes, the field value is sent to a storage node of a preset database, otherwise, another field value is selected from the field name list and the value list, and the process of the step 210 to the step 260 is repeated until the judgment of all the field values is completed.
optionally, as shown in fig. 4, if the operation type of the analyzed data change log is UPDATE, step 140 specifically includes:
step 410, obtaining the name and the field modification value of the table stored in the data change log;
step 420, acquiring field names and field values appearing in the field modification values to spell out clauses of preset types;
step 430, inquiring the distribution key information of the table;
Step 440, filtering out the distribution key field of the table in the database from the field value and converting the distribution key field into a preset condition;
step 450, synchronize the field values to the database.
Specifically, the name of the table is obtained according to the object _ name stored in the data change log, and a clause of a preset type is spelled according to a field name and a field value appearing in a field modification value (new _ value _ list) stored in the data change log, where the clause of the preset type may be a set clause. And inquiring the distribution key information of the table from the data cache, and filtering the distribution key field of the table in the distributed database from the field value to form a preset condition, wherein the preset condition can be a where condition. And synchronizing to the middleware of the distributed database according to the granularity of the SQL statements so that the middleware sends the statements needing to be synchronized to the specified database storage nodes according to the distribution attributes.
Optionally, as shown in fig. 5, if the operation type of the parsed data change log is DELETE, step 140 specifically includes:
Step 510, obtaining a name of a table stored in the data change log;
Step 520, inquiring the distribution key information of the table;
step 530, filtering out the distribution key field of the table in the database from the field value and converting the distribution key field into a preset condition;
step 540, synchronize the field values to the database.
specifically, the name of the table is obtained according to the object _ name stored in the data change log, the distribution key information of the table is queried from the data cache, and the distribution key field of the table in the distributed database is filtered from the field value to form a preset condition, where the preset condition may be a where condition. And synchronizing to the middleware of the distributed database according to the granularity of the SQL statements so that the middleware sends the statements needing to be synchronized to the specified database storage nodes according to the distribution attributes.
Optionally, as shown in fig. 6, if the operation type of the analyzed data change log is not INSERT, UPDATE, DELETE, the method for incrementally synchronizing data of a distributed database in this embodiment further includes:
Step 610, judging whether the change data is synchronized to the database; if not, go to step 620;
Step 620, uploading an alarm;
Step 630, classifying the reported alarm types;
step 640, judging whether the alarm is a serious alarm; if yes, go to step 650;
Step 650, generating an alarm prompt.
Specifically, when the incremental data synchronization executed by the distributed database fails, the distributed database performs alarm processing, checks the uploaded alarms, and performs classification processing on the alarm types. If the alarm is a common alarm, the alarm is not processed, and if the alarm is a serious alarm, an alarm prompt is generated and manual intervention is performed.
Optionally, as shown in fig. 7, after step 140, the incremental data synchronization method for a distributed database of this embodiment further includes:
Step 710, perfecting the SQL statement according to distribution key information and distribution attributes of the table related to the SQL statement;
step 720, sending the data to a preset data storage node through the completed SQL statement so that the data storage node executes the completed SQL statement and obtains an execution result;
step 730, receiving and verifying the execution result.
Specifically, after the changed data is synchronized to the database, the SQL statement is perfected according to distribution key information and distribution attributes of a table related in the SQL statement, and the data is distributed to a storage node of a designated database according to distribution characteristics, so that the storage node executes the SQL statement, receives an execution result of the storage node on the SQL statement, and performs data synchronization consistency check on the execution result.
the incremental data synchronization method for the distributed database of the embodiment includes acquiring change data in the DBMS, converting the change data into data change logs, analyzing the data change logs to generate corresponding SQL statements, synchronizing the change data to the distributed database through the SQL statements and according to distribution key information, capturing changes of the DBMS data, synchronizing the changes to the distributed database in real time, and generating SQL synchronization data according to a distribution strategy.
second embodiment
Fig. 8 is a schematic diagram of a hardware architecture of a device according to a second embodiment of the present invention. In fig. 8, the apparatus includes: a memory 810, a processor 820, and a distributed database incremental data synchronization program 830 stored on the memory 810 and executable on the processor 820. In this embodiment, the distributed database incremental data synchronization program 830 includes a series of computer program instructions stored in the memory 810, which when executed by the processor 820, can implement the distributed database incremental data synchronization operation according to embodiments of the present invention. In some embodiments, distributed database incremental data synchronization program 830 may be divided into one or more modules based on the particular operations implemented by the portions of the computer program instructions. As shown in fig. 9, the distributed database incremental data synchronization program 830 includes: the system comprises an acquisition module 910, a change data conversion module 920, an analysis module 930, a synchronization module 940, a judgment module 950, an alarm processing module 960, a sending module 970 and a verification module 980. Wherein the content of the first and second substances,
an obtaining module 910, configured to obtain change data in the DBMS.
specifically, the obtaining module 910 obtains change data of all DMLs on the DBMS.
and a change data conversion module 920, configured to convert the change data into a data change log.
specifically, the operation types of the data change log include INSERT INSERT, UPDATE UPDATE, and DELETE DELETE.
The parsing module 930 is configured to parse the data change log to generate a corresponding SQL statement.
Specifically, the parsing module 930 parses the data change log of the user, which includes the operation type, the data before and after the change, and the like. And spelling out partial SQL according to the analysis result.
And a synchronization module 940, configured to synchronize the change data to the distributed database according to the distribution information of the distributed database table through the SQL statement.
specifically, the synchronization module 940 synchronizes the data to the database, which may be a distributed database, through the generated SQL statement and according to the distribution key information.
Optionally, in this embodiment, a registry is used to record names (abbreviated as table names) of all tables of the data change log to be captured, the table is called a synchronization source table, and information such as a table name of a change, a user name to which the table belongs, and an operation type is recorded in detail in the synchronization source table. After the data change log is captured, the DML statement is restored through data analysis of the data change log. Different operation types and different modes for restoring the DML statement are adopted, and for the INSERT statement, the SQL statement to be executed can be restored according to the operation type, the table name and the modified field value; for UPDATE and DELETE statements, with the where condition in the statement, if all fields are pieced together one by one into the where clause, it not only makes the statement very long, for example: some tables of the service may have hundreds of fields, and due to the limitation of the distributed database, the index cannot be used during the execution on the distributed database side, so that the synchronization of the UPDATE and DELETE statements is optimized based on the distribution attribute and the distribution node information of the distributed database in consideration of the characteristics of the distributed database.
in the embodiment, the change of the DBMS data is captured, the change data log file is analyzed, the SQL statement is generated, the data is synchronized according to the distribution strategy based on the characteristics of the distributed database, the data synchronization speed is improved, meanwhile, the distributed database is used for batch processing in the synchronization process, the performance is improved, the data correctness is verified through the script after the data synchronization is completed, and the data consistency is ensured.
if the analyzed operation type of the data change log is INSERT, the synchronization module 940 is specifically configured to:
obtaining the name and the field modification value of the table stored in the data change log;
acquiring a field name list and a field value list according to the field modification value;
Selecting a field value from the field name list and the field value list;
Judging whether the field value is NULL or not; if yes, the field values are not synchronized, and if not, the field values are synchronized.
specifically, the name of the table is obtained according to the object _ name stored in the data change log, and the field name list and the field value list are obtained according to the field modification value (new _ value _ list) stored in the data change log, when the field value is judged to be NULL, it is indicated that the source statement is not assigned, the field should not appear in the field list of the insert statement, the field value does not need to be synchronized, and otherwise, the field value needs to be synchronized.
Optionally, the synchronization module 940 is further configured to: judging whether the field value is the last field in the field name list and the field value list; if yes, the field value is sent to a preset storage node, and if not, another field value is selected from the field name list and the field value list again.
specifically, after the field value is judged to be NULL and not to be synchronized, and the field value is judged not to be NULL and to be synchronized, whether the field value is the last field in the field name list and the value list needs to be judged, if yes, the field value is sent to a storage node of a preset database, otherwise, another field value is selected from the field name list and the value list, and the process of the step 210 to the step 260 is repeated until the judgment of all the field values is completed.
Optionally, if the operation type of the analyzed data change log is UPDATE, the synchronization module 940 is specifically configured to:
obtaining the name and the field modification value of the table stored in the data change log;
acquiring field names and field values appearing in the field modification values to spell out clauses of preset types;
querying distribution key information of the table;
Filtering out the distribution key field of the table in the database from the field value and converting the distribution key field into a preset condition;
synchronizing the field values to the database.
specifically, the name of the table is obtained according to the object _ name stored in the data change log, and a clause of a preset type is spelled according to a field name and a field value appearing in a field modification value (new _ value _ list) stored in the data change log, where the clause of the preset type may be a set clause. And inquiring the distribution key information of the table from the data cache, and filtering the distribution key field of the table in the distributed database from the field value to form a preset condition, wherein the preset condition can be a where condition. And synchronizing to the middleware of the distributed database according to the granularity of the SQL statements so that the middleware sends the statements needing to be synchronized to the specified database storage nodes according to the distribution attributes.
Optionally, if the operation type of the analyzed data change log is DELETE, the synchronization module 940 is specifically configured to:
Obtaining the name of a table stored in the data change log;
querying distribution key information of the table;
Filtering out the distribution key field of the table in the database from the field value and converting the distribution key field into a preset condition;
synchronizing the field values to the database.
Specifically, the name of the table is obtained according to the object _ name stored in the data change log, the distribution key information of the table is queried from the data cache, and the distribution key field of the table in the distributed database is filtered from the field value to form a preset condition, where the preset condition may be a where condition. And synchronizing to the middleware of the distributed database according to the granularity of the SQL statements so that the middleware sends the statements needing to be synchronized to the specified database storage nodes according to the distribution attributes.
A determining module 950, configured to determine whether the change data is synchronized to the database when the operation type of the analyzed data change log is not INSERT, UPDATE, or DELETE.
An alarm processing module 960, configured to upload an alarm and perform classification processing on the reported alarm types when the determining module 950 determines that synchronization of the change data fails;
The judging module 950 is further configured to judge whether the alarm is a serious alarm; if so, the alarm processing module 960 is further configured to generate an alarm prompt.
Specifically, when the incremental data synchronization performed by the distributed database fails, the alarm processing module 960 performs alarm processing, checks the uploaded alarms, and classifies the alarm types, in this embodiment, the alarm types are classified into general alarms and serious alarms. If the alarm is a common alarm, the alarm is not processed, and if the alarm is a serious alarm, an alarm prompt is generated and manual intervention is performed.
The sending module 970 is configured to perfect the SQL statement according to the distribution key information and the distribution attributes of the table related to the SQL statement, and send data to a preset data storage node through the finished SQL statement, so that the data storage node executes the finished SQL statement and obtains an execution result;
a checking module 980 for receiving and checking the execution result.
specifically, after the changed data is synchronized to the database, the SQL statement is completed according to the distribution key information and the distribution attributes of the table related in the SQL statement, the sending module 970 distributes the data to the storage node of the designated database according to the distribution characteristics, so that the storage node executes the SQL statement, and the checking module 980 receives the execution result of the storage node on the SQL statement and performs data synchronization consistency checking on the execution result.
The device of this embodiment obtains the change data in the DBMS through the obtaining module 910, the change data conversion module 920 converts the change data into a data change log, the parsing module 930 parses the data change log to generate a corresponding SQL statement, the synchronization module 940 synchronizes the change data to the distributed database through the SQL statement and according to the distribution key information, and can capture the change of the DBMS data and synchronize the change to the distributed database in real time, and generate SQL synchronization data according to the distribution policy.
third embodiment
the embodiment of the invention also provides a computer readable storage medium. The computer-readable storage medium herein stores one or more programs. Among other things, computer-readable storage media may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, a hard disk, or a solid state disk; the memory may also comprise a combination of memories of the kind described above. When one or more programs in the computer readable storage medium are executable by one or more processors, the distributed database incremental data synchronization method provided by the first embodiment is implemented.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
the above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
while the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method for incremental data synchronization of a distributed database, the method comprising the steps of:
acquiring change data in a database management system (DBMS);
Converting the change data into a data change log;
Analyzing the data change log to generate a corresponding Structured Query Language (SQL) statement;
And synchronizing the change data to the distributed database through the SQL statement according to the distribution information of the distributed database table.
2. the incremental data synchronization method for distributed databases of claim 1, wherein if the operation type of the parsed data change log is INSERT, synchronizing the change data to the distributed database comprises:
Obtaining the name and the field modification value of the table stored in the data change log;
acquiring a field name list and a field value list according to the field modification value;
Selecting a field value from the field name list and the field value list;
judging whether the field value is NULL or not;
If so, not synchronizing the field values;
if not, synchronizing the field value.
3. The method of distributed database delta data synchronization as recited in claim 2, wherein after determining that the field values are not synchronized or that the field values are synchronized, the method further comprises:
Judging whether the field value is the last field in the field name list and the field value list;
If yes, the field value is issued to a preset storage node;
if not, another field value is selected from the field name list and the field value list again.
4. the incremental data synchronization method for distributed databases of claim 1, wherein if the operation type of the parsed data change log is UPDATE, then synchronizing the change data to the distributed database comprises:
obtaining the name and the field modification value of the table stored in the data change log;
acquiring field names and field values appearing in the field modification values to spell out clauses of preset types;
Querying distribution key information of the table;
Filtering out the distribution key field of the table in the database from the field value and converting the distribution key field into a preset condition;
Synchronizing the field values to the database.
5. the incremental data synchronization method for distributed databases of claim 1, wherein if the operation type of the parsed data change log is DELETE, then synchronizing the change data to the distributed database comprises:
obtaining the name of a table stored in the data change log;
querying distribution key information of the table;
filtering out the distribution key field of the table in the database from the field value and converting the distribution key field into a preset condition;
Synchronizing the field values to the database.
6. the incremental data synchronization method for distributed databases as claimed in claim 1, wherein if the operation type of the parsed data change log is not INSERT, UPDATE, DELETE, the method further comprises:
judging whether the change data is synchronized to the database;
And if not, uploading an alarm.
7. The incremental data synchronization method for distributed databases as claimed in claim 6, wherein after the uploading of the alert, the method further comprises:
classifying the reported alarm types;
Judging whether the alarm is a serious alarm or not;
If yes, an alarm prompt is generated.
8. the incremental data synchronization method for distributed databases of claim 1, wherein after synchronizing the change data to the distributed databases, the method further comprises:
Perfecting the SQL statement according to distribution key information and distribution attributes of the table related to the SQL statement;
Sending data to a preset data storage node through the completed SQL statement so that the data storage node executes the completed SQL statement and obtains an execution result;
and receiving and checking the execution result.
9. an apparatus, comprising a processor and a memory;
The processor is configured to execute a distributed database incremental data synchronization program stored in the memory to implement the method of any of claims 1-8.
10. a computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more programs which are executable by one or more processors to implement the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533666.1A CN110543513B (en) | 2018-05-29 | 2018-05-29 | Incremental data synchronization method, equipment and storage medium for distributed database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810533666.1A CN110543513B (en) | 2018-05-29 | 2018-05-29 | Incremental data synchronization method, equipment and storage medium for distributed database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110543513A true CN110543513A (en) | 2019-12-06 |
| CN110543513B CN110543513B (en) | 2024-03-15 |
Family
ID=68701585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810533666.1A Active CN110543513B (en) | 2018-05-29 | 2018-05-29 | Incremental data synchronization method, equipment and storage medium for distributed database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110543513B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143470A (en) * | 2019-12-27 | 2020-05-12 | 中国银行股份有限公司 | Cross-platform database data synchronization comparison method and device |
| CN111427898A (en) * | 2020-03-16 | 2020-07-17 | 上海爱数信息技术股份有限公司 | Continuous data protection system and method based on analysis of Oracle log |
| CN111597257A (en) * | 2020-05-09 | 2020-08-28 | 远光软件股份有限公司 | Database synchronization method and device, storage medium and terminal |
| CN112231396A (en) * | 2020-09-15 | 2021-01-15 | 山东浪潮通软信息科技有限公司 | Method for incrementally synchronizing data across databases |
| CN112395360A (en) * | 2020-12-01 | 2021-02-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN113672639A (en) * | 2021-08-27 | 2021-11-19 | 成都长城开发科技有限公司 | Method, system, equipment and storage medium for comparing structures of multi-type database tables |
| CN113792066A (en) * | 2021-09-23 | 2021-12-14 | 中国农业银行股份有限公司 | Dynamic analysis scheduling method and device for batch SQL |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187888A (en) * | 2007-12-11 | 2008-05-28 | 浪潮电子信息产业股份有限公司 | Method for coping database data in heterogeneous environment |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| US20140324778A1 (en) * | 2013-04-26 | 2014-10-30 | Netapp, Inc. | Hierarchical Coherency Log for Managing a Distributed Data Storage System |
| EP2891994A1 (en) * | 2013-11-04 | 2015-07-08 | Guangdong Electronics Industry Institute Ltd. | Method for achieving automatic synchronization of multisource heterogeneous data resources |
| CN105005618A (en) * | 2015-07-21 | 2015-10-28 | 杭州合众数据技术有限公司 | Data synchronization method and system among heterogeneous databases |
| CN105447054A (en) * | 2014-09-25 | 2016-03-30 | 中兴通讯股份有限公司 | Data synchronization method, device and system for database |
| CN105824865A (en) * | 2015-12-09 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed type database data synchronization system |
| CN106250543A (en) * | 2016-08-10 | 2016-12-21 | 深圳市彬讯科技有限公司 | A kind of automation data inquiry synchronous storage method |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| US20180081956A1 (en) * | 2013-11-04 | 2018-03-22 | Guangdong Electronics Industry Institute Ltd. | Method for automatically synchronizing multi-source heterogeneous data resources |
| KR101825910B1 (en) * | 2016-08-31 | 2018-03-22 | 국방과학연구소 | Heterogeneous database synchronization system and method using neutral data based on redo log analysis |
| US20180137187A1 (en) * | 2016-11-11 | 2018-05-17 | International Business Machines Corporation | Copying data changes to a target database |
-
2018
- 2018-05-29 CN CN201810533666.1A patent/CN110543513B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101187888A (en) * | 2007-12-11 | 2008-05-28 | 浪潮电子信息产业股份有限公司 | Method for coping database data in heterogeneous environment |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| US20140324778A1 (en) * | 2013-04-26 | 2014-10-30 | Netapp, Inc. | Hierarchical Coherency Log for Managing a Distributed Data Storage System |
| EP2891994A1 (en) * | 2013-11-04 | 2015-07-08 | Guangdong Electronics Industry Institute Ltd. | Method for achieving automatic synchronization of multisource heterogeneous data resources |
| US20180081956A1 (en) * | 2013-11-04 | 2018-03-22 | Guangdong Electronics Industry Institute Ltd. | Method for automatically synchronizing multi-source heterogeneous data resources |
| CN103761318A (en) * | 2014-01-27 | 2014-04-30 | 中国工商银行股份有限公司 | Method and system for data synchronization of relational heterogeneous databases |
| CN105447054A (en) * | 2014-09-25 | 2016-03-30 | 中兴通讯股份有限公司 | Data synchronization method, device and system for database |
| CN105005618A (en) * | 2015-07-21 | 2015-10-28 | 杭州合众数据技术有限公司 | Data synchronization method and system among heterogeneous databases |
| CN105824865A (en) * | 2015-12-09 | 2016-08-03 | 广东亿迅科技有限公司 | Distributed type database data synchronization system |
| CN106250543A (en) * | 2016-08-10 | 2016-12-21 | 深圳市彬讯科技有限公司 | A kind of automation data inquiry synchronous storage method |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| KR101825910B1 (en) * | 2016-08-31 | 2018-03-22 | 국방과학연구소 | Heterogeneous database synchronization system and method using neutral data based on redo log analysis |
| US20180137187A1 (en) * | 2016-11-11 | 2018-05-17 | International Business Machines Corporation | Copying data changes to a target database |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143470A (en) * | 2019-12-27 | 2020-05-12 | 中国银行股份有限公司 | Cross-platform database data synchronization comparison method and device |
| CN111427898A (en) * | 2020-03-16 | 2020-07-17 | 上海爱数信息技术股份有限公司 | Continuous data protection system and method based on analysis of Oracle log |
| CN111597257A (en) * | 2020-05-09 | 2020-08-28 | 远光软件股份有限公司 | Database synchronization method and device, storage medium and terminal |
| CN112231396A (en) * | 2020-09-15 | 2021-01-15 | 山东浪潮通软信息科技有限公司 | Method for incrementally synchronizing data across databases |
| CN112395360A (en) * | 2020-12-01 | 2021-02-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN112395360B (en) * | 2020-12-01 | 2023-06-23 | 中国联合网络通信集团有限公司 | Data synchronization method, device, apparatus and medium based on non-relational database |
| CN113672639A (en) * | 2021-08-27 | 2021-11-19 | 成都长城开发科技有限公司 | Method, system, equipment and storage medium for comparing structures of multi-type database tables |
| CN113672639B (en) * | 2021-08-27 | 2024-04-26 | 成都长城开发科技股份有限公司 | Multi-type database table structure comparison method, system, equipment and storage medium |
| CN113792066A (en) * | 2021-09-23 | 2021-12-14 | 中国农业银行股份有限公司 | Dynamic analysis scheduling method and device for batch SQL |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110543513B (en) | 2024-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110543513B (en) | Incremental data synchronization method, equipment and storage medium for distributed database | |
| US11157512B2 (en) | Method and system for replicating data to heterogeneous database and detecting synchronization error of heterogeneous database through SQL packet analysis | |
| CN108509658B (en) | XML file parsing method and device | |
| US7418461B2 (en) | Schema conformance for database servers | |
| CN109739867B (en) | Industrial metadata management method and system | |
| WO2020253399A1 (en) | Log classification rule generation method, device, apparatus, and readable storage medium | |
| CN105912594B (en) | SQL statement processing method and system | |
| CN107506383B (en) | Audit data processing method and computer equipment | |
| CN112685433A (en) | Metadata updating method and device, electronic equipment and computer-readable storage medium | |
| CN110659282A (en) | Data route construction method and device, computer equipment and storage medium | |
| CN111625540A (en) | Method and device for verifying data synchronization integrity of relational database | |
| CN111930842B (en) | Data checking method and device | |
| CN111046100A (en) | Method and system for synchronizing relational database to non-relational database | |
| CN105719072B (en) | System and method for associating multi-segment component transactions | |
| CN111221967A (en) | Language data classification storage system based on block chain architecture | |
| CN115658815A (en) | CDC (control data center) -based data synchronization method | |
| CN113221528B (en) | Automatic generation and execution method of clinical data quality evaluation rule based on openEHR model | |
| CN114911872A (en) | Intranet and extranet data synchronization method, device and system, extranet server and storage medium | |
| CN112395292B (en) | Data feature extraction and matching method and device | |
| CN111143329B (en) | Data processing method and device | |
| CN112860637A (en) | Method and system for processing log based on audit strategy | |
| CN110932896A (en) | Method, device and equipment for creating log inverted index and readable storage medium | |
| CN112597207B (en) | Metadata management system | |
| Jandaeng | Comparison of RDBMS and document oriented database in audit log analysis | |
| CN116701441A (en) | Database slow query detection method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220112 Address after: 100176 602, floor 6, building 6, courtyard 10, KEGU 1st Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing (Yizhuang group, high-end industrial area of Beijing Pilot Free Trade Zone) Applicant after: Jinzhuan Xinke Co.,Ltd. Address before: 518000 Zhongnan communication tower, South China Road, Nanshan District high tech Industrial Park, Shenzhen, Guangdong Applicant before: ZTE Corp. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |