CN110737720A - DB2 database data synchronization method, device and system - Google Patents
DB2 database data synchronization method, device and system Download PDFInfo
- Publication number
- CN110737720A CN110737720A CN201910842876.3A CN201910842876A CN110737720A CN 110737720 A CN110737720 A CN 110737720A CN 201910842876 A CN201910842876 A CN 201910842876A CN 110737720 A CN110737720 A CN 110737720A
- Authority
- CN
- China
- Prior art keywords
- synchronization
- library
- full
- source
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses DB2 database data synchronization methods, devices and systems, and the method comprises the steps of obtaining data synchronization requests for a source library to be synchronized and a corresponding target library to be synchronized, carrying out full synchronization on the target library according to data of the source library if the requests are full synchronization requests, obtaining log data generated by the source library in real time and analyzing the log data if the requests are incremental synchronization requests, carrying out incremental synchronization on the target library according to analysis results, achieving full and incremental synchronization of a DB2 database, enabling the incremental synchronization mode of the application to not invade the source library to not affect service, achieving sub-library and sub-table scene synchronization and monitoring and coordination of sub-library and sub-table synchronization through expanding clusters, enabling mapping relation configuration of the source library and the target library through a middleware, achieving data synchronization from the source library M to the target library N, meeting more complex synchronization scenes, and entering steps, wherein the full and incremental synchronization processes support breakpoint continuous transmission.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to a data synchronization method, device and system for DB2 databases.
Background
Data synchronization is not fresh concepts in the field of databases, many companies have their own open-source or closed-source data synchronization products, such as golden gate of Oracle, keyle of Pentaho, CDC of IBM, canal of domestic ali, debezium of open source, and confluent jdbc connector, however, many existing database data synchronization products only support mysql synchronization and cannot achieve incremental synchronization of db2 data, furthermore, synchronization tools such as golden gate and CDC can only achieve single source synchronization and cannot achieve data synchronization of scenes in which the source libraries are partitioned tables, and further, steps make the existing synchronization products unable to achieve breakpoint resuming of the synchronization process.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides DB2 database data synchronization methods, devices and systems, the full-scale and incremental synchronization of a DB2 database can be realized, and the incremental synchronization mode of the invention has no invasion to a source library and cannot influence the service.
The embodiment of the invention provides the following specific technical scheme:
, the invention provides DB2 database data synchronization methods, the method includes:
acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
if the request is a full-scale synchronization request, performing full-scale synchronization on the target library according to the data of the source library;
and if the request is an incremental synchronization request, acquiring and analyzing the log data generated by the source library in real time, and performing incremental synchronization on the target library according to an analysis result.
Preferably, the method further comprises:
determining the database dividing information of the source database and the corresponding database dividing information of the target database;
and establishing communication connection between the sub-libraries of the source library and the sub-libraries of the target library according to the middleware.
Preferably, the method further comprises:
creating a cluster management node, and monitoring the synchronous process from the sub-library of the source library to the sub-library of the target library in real time according to the cluster management node;
and if the synchronization process from any of the source library to the target library is abnormal, stopping the synchronization process from the rest of the source library to the target library.
Preferably, the acquiring and analyzing the log data generated by the source library in real time specifically includes:
calling a jni method at a Java end, and entering a program c;
analyzing data synchronization request parameters transmitted by the Java terminal in the c program; wherein the data synchronization request parameter comprises a database location;
determining an initial position of incremental synchronization, and acquiring the log data generated after the initial position;
analyzing the log data according to a predefined DB2 log protocol to obtain an analysis result;
and returning the analysis result to the Java end according to the callback Java method.
Preferably, the method further comprises:
monitoring the incremental synchronization process;
if abnormal interruption occurs in the increment synchronization, recording and storing the position of the latest times of increment synchronization to the target library in real time before the abnormal interruption;
and determining the position as an abnormal interruption point, and performing incremental synchronization again according to the abnormal interruption point to realize breakpoint continuous transmission in the incremental synchronization process.
Preferably, the performing incremental synchronization on the target library to be synchronized according to the analysis result specifically includes:
judging the type of the analysis result;
when the analysis result is a commit or rollback operation, writing the commit or rollback operation into a transaction queue of the same ;
judging the sequence of the transaction for processing the commit or rollback operation in the transaction queue;
and writing the corresponding submission or rollback operation into the target library in sequence according to the sequence.
Preferably, the method further comprises:
monitoring the full-scale synchronization process;
recording and storing the table name of the source library in real time when full synchronization is carried out;
and if abnormal interruption occurs in the full synchronization, positioning according to the table name of the source library recorded and stored in real time to obtain an abnormal interruption point, and performing the full synchronization again according to the abnormal interruption point to realize the breakpoint continuous transmission of the table level in the full synchronization process.
Preferably, the method further comprises:
sequencing every rows of the table of the source library, and acquiring primary key values of every rows after sequencing;
recording and storing primary key values of every rows of the table of the source library in real time when the full-scale synchronization is carried out;
and if the full synchronization has abnormal interruption, positioning according to primary key values of every rows of the table of the source library recorded and stored in real time to obtain an abnormal interruption point, and performing the full synchronization again according to the abnormal interruption point to realize the breakpoint continuous transmission of the record level in the full synchronization process.
In a second aspect, the present invention provides DB2 database data synchronization devices, the devices comprising:
the acquisition module is used for acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
, a synchronization module, configured to perform full synchronization on the target repository according to the data of the source repository if the request is a full synchronization request;
and the second synchronization module is used for acquiring and analyzing the log data generated by the source library in real time if the request is an incremental synchronization request, and performing incremental synchronization on the target library according to an analysis result.
Preferably, the apparatus further comprises:
the configuration module is used for determining the database dividing information of the source database and the corresponding database dividing information of the target database;
and establishing communication connection between the sub-libraries of the source library and the sub-libraries of the target library according to the middleware.
Preferably, the apparatus further comprises:
the monitoring module is used for creating a cluster management node and monitoring a synchronous process from the sub-library of the source library to the sub-library of the target library in real time according to the cluster management node;
and if the synchronization process from any of the source library to the target library is abnormal, stopping the synchronization process from the rest of the source library to the target library.
Preferably, the second synchronization module includes:
the log analysis module is used for calling the jni method at the Java end and entering a program c;
analyzing data synchronization request parameters transmitted by the Java terminal in the c program; wherein the data synchronization request parameter comprises a database location;
determining an initial position of incremental synchronization, and acquiring the log data generated after the initial position;
analyzing the log data according to a predefined DB2 log protocol to obtain an analysis result;
and returning the analysis result to the Java end according to the callback Java method.
Preferably, the monitoring module is specifically configured to monitor the incremental synchronization process;
the device further comprises:
the recording module is used for recording and storing a position when the latest times of increment synchronization is carried out to the target library before the abnormal interruption in real time if the increment synchronization has the abnormal interruption;
and the abnormal process processing module is used for determining the site as an abnormal interruption point and carrying out increment synchronization again according to the abnormal interruption point so as to realize breakpoint continuous transmission in the increment synchronization process.
Preferably, the second synchronization module further includes:
the concurrent transaction processing module is used for judging the type of the analysis result;
when the analysis result is a commit or rollback operation, writing the commit or rollback operation into a transaction queue of the same ;
judging the sequence of the transaction for processing the commit or rollback operation in the transaction queue;
and writing the corresponding submission or rollback operation into the target library in sequence according to the sequence.
Preferably, the monitoring module is further configured to monitor the full-scale synchronization process;
the recording module is also used for recording and storing the table name of the source library in real time when full synchronization is carried out;
and the abnormal process processing module is also used for positioning according to the table name of the source library which is recorded and stored in real time to obtain an abnormal interruption point if the full synchronization has abnormal interruption, and performing the full synchronization again according to the abnormal interruption point to realize the breakpoint continuous transmission of the table level in the full synchronization process.
Preferably, the apparatus further comprises:
the sorting module is used for sorting every rows of the table of the source library and acquiring primary key values of every rows after sorting;
the recording module is further used for recording and storing primary key values of each rows of the table of the source library in real time when the full-scale synchronization is carried out;
and the abnormal process processing module is also used for positioning according to the primary key values of every rows of the table of the source library recorded and stored in real time to obtain an abnormal interruption point if the full synchronization has abnormal interruption, and performing the full synchronization again according to the abnormal interruption point to realize the breakpoint continuous transmission of the record level in the full synchronization process.
In a third aspect, the present invention provides computer systems, comprising:
or more processors, and
memory associated with the or more processors for storing program instructions that when read executed by the or more processors perform the following:
acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
if the request is a full-scale synchronization request, performing full-scale synchronization on the target library according to the data of the source library;
and if the request is an incremental synchronization request, acquiring and analyzing the log data generated by the source library in real time, and performing incremental synchronization on the target library according to an analysis result.
The embodiment of the invention has the following beneficial effects:
1. the invention can realize the full and incremental synchronization of the DB2 database, and when the incremental synchronization is carried out, the incremental data is obtained by obtaining the log data generated by the source library and analyzing the log data, and the method has no invasion to the source library and does not influence the service;
2. when the increment is synchronized, the log analysis is carried out through the program c, the analysis efficiency is higher, and the synchronization delay is low;
3. according to the method, the synchronization of the sub-database and sub-table scenes and the monitoring and coordination of the sub-database and sub-table synchronization can be realized by expanding the cluster, and meanwhile, the mapping relation configuration is carried out on the source database and the target database through the middleware, so that the data synchronization from the source database M to the target database N can be realized, and a more complex synchronization scene is met;
4. the invention supports breakpoint continuous transmission in the full and incremental synchronization processes.
Of course, the present invention does not necessarily require to satisfy all of the above effects at the same time, but only .
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow chart of DB2 database data synchronization methods provided in embodiment of the present application;
FIG. 2 is a flowchart illustrating the type determination of the parsing result and the synchronous writing into the target library when the source library is a concurrent transaction according to embodiment of the present application;
FIG. 3 is a schematic structural diagram of a data synchronization apparatus for DB2 databases according to a second embodiment of the present application;
FIG. 4 is a diagram of a computer system architecture provided in a third embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments, but not all embodiments, of the present invention.
Example
As shown in fig. 1, the present application provides DB2 database data synchronization methods, comprising the following steps:
and S22, acquiring data synchronization requests aiming at the source library to be synchronized and the corresponding target library to be synchronized.
In the scheme, the source library to be synchronized is a DB2 database, the target library can be a DB2 database, or other relational databases such as mysql, oracle, sybase and the like, in addition, the source library and the target library to be synchronized can be or more, and before the step, the source library and the target library to be synchronized are also required to be configured in a related manner to realize the communication connection of the source library and the target library.
Taking a plurality of source libraries and a plurality of target libraries as an example, the process of configuring the source libraries and the target libraries is specifically described, and comprises the following steps:
1. determining the database dividing information of a source database and the corresponding database dividing information of a target database;
2. and establishing communication connection between the sub-libraries of the source library and the sub-libraries of the target library according to the middleware.
In the scheme, the middleware is used as a connecting bridge between a source library and a target library and is used for receiving data from the branch libraries of the source library and distributing the data of the branch libraries of the source library to the branch libraries of the target library according to routing rules of the middleware, the middleware can be Drds, Mycat or any other software which can realize the functions, and the selection of the middleware is not limited.
And S24, if the request is a full-scale synchronization request, performing full-scale synchronization on the target library according to the data of the source library.
Specifically, when the request is a full-scale synchronization request, the table to be synchronized in the source repository is directly synchronized to the target repository. It should be noted that, because the foregoing has implemented the configuration connection from multiple source libraries to multiple target libraries, the present solution may also implement the full data synchronization from multiple sub-libraries of the source library to multiple sub-libraries of the target library, and specifically may include the following steps:
1. acquiring table data in sub-libraries of a source library;
2. and distributing and writing the table data in the sub-libraries of the source library into the sub-libraries of the target library according to the routing rule.
In addition, in order to coordinate and manage the data synchronization process of the sub-database and sub-table of the source database and prevent the mutual influence caused by the error of the sub-database and sub-table in the data synchronization process, the scheme further comprises the following steps:
1. creating a cluster management node, and monitoring the synchronous process from the sub-library of the source library to the sub-library of the target library in real time according to the cluster management node;
2. and if the synchronization process from any of the source library to the target library is abnormal, stopping the synchronization process from the rest of the source library to the target library.
Illustratively, if the source library has three sub-libraries, the target library has two sub-libraries, db2sync processes are deployed on the three sub-libraries of the source library respectively, the db2sync processes of the three sub-libraries are included into groups, and according to the created cluster management node (active-standby model based on zookeeper), when any db2sync process of the three sub-libraries is abnormal, the db2sync processes of the other two sub-libraries are stopped.
In addition, when the full synchronization process is abnormal, the scheme can also realize breakpoint continuous transmission. The method specifically comprises the following steps:
1. monitoring a full-scale synchronization process;
2. recording and storing the table name of the source library in real time when full synchronization is carried out;
3. and if the full synchronization has abnormal interruption, positioning according to the table name of the source library recorded and stored in real time to obtain an abnormal interruption point, and performing the full synchronization again according to the abnormal interruption point.
The breakpoint continuous transmission depends on recording and storing of breakpoint information, the existing synchronization scheme cannot achieve breakpoint continuous transmission in a full synchronization stage, and the scheme can achieve breakpoint continuous transmission in a table level in the full synchronization process through recording and storing a table name of a source base in the full synchronization process in real time, so that the synchronized tables do not need to be synchronized, and the synchronized tables which are in progress when abnormal conditions occur are resynchronized.
In addition, steps further, the scheme can also realize breakpoint transmission of the record level in the full-volume synchronization process.
The method specifically comprises the following steps:
1. sequencing every rows of the table of the source library, and acquiring primary key values of every rows after sequencing;
2. monitoring a full-scale synchronization process;
3. recording and storing the table name of the source library and the primary key value of each rows of the table in real time when full synchronization is carried out;
4. and if the full synchronization has abnormal interruption, positioning according to the table name of the source library recorded and stored in real time and the primary key value of each rows of the table to obtain an abnormal interruption point, and performing the full synchronization again according to the abnormal interruption point.
Through the steps, when an abnormal condition occurs in the full-scale synchronization process, all records (rows) in the table are not synchronized like the breakpoint continuous transmission at the table level, and the synchronization is started from a certain row in the table, so that the positioning is more accurate.
And S26, if the request is an incremental synchronization request, acquiring and analyzing the log data generated by the source library in real time, and performing incremental synchronization on the target library according to the analysis result.
In particular, incremental synchronization is used to synchronize the incremental data generated from the full start to the end of the task. The method for acquiring and analyzing the log data generated by the source library in real time comprises the following steps:
1. calling a jni method at a Java end, and entering a program c;
the Java calls the bottom layer c program through the jni method, so that the analysis efficiency is higher, and the synchronization delay is reduced.
2. Analyzing data synchronization request parameters transmitted by a Java end in a program c; wherein, the data synchronization request parameter comprises a database site;
in addition, the data synchronization request parameters also include a source library name, a schema name, a user name, a password and the like.
3. Determining an initial position of incremental synchronization, and acquiring log data generated after the initial position;
specifically, the step may include the steps of:
(1) judging whether an incremental synchronization start site is appointed or not;
(2) if not, acquiring the latest database site as an initial site;
(3) if the database is specified, taking the specified database site as an initial site;
(4) and acquiring log data generated after the starting site.
When the starting site is not specified, the latest database site can be obtained by calling a DB2ReadLog (iCallAction ═ DB2READLOG _ QUERY) function; when an initial site is appointed, whether log data are READ or not can be judged by calling a DB2ReadLog (iCallAction ═ DB2READLOG _ READ) function and returning a logRecWritten field of an object (DB2 ReadLogInfStrect type), and when the log data are not READ, the DB2ReadLog (iCallAction ═ DB2READLOG _ READ) function is called again after the object is dormant according to set time (such as 100ms) until the log data are obtained; in addition, the acquired log data can be stored in a pre-allocated memory space.
4. Analyzing the log data according to a predefined DB2 log protocol to obtain an analysis result;
specifically, each field is analyzed according to a predefined DB2 log protocol to obtain an analysis result, wherein the fields comprise a transaction number, an operation type, a table id, a table schema id, operation data and the like.
Illustratively, the result obtained after resolving insertion statements (insert statements) is as follows:
5. and returning the analysis result to the Java end according to the callback Java method.
In addition, the incremental synchronization of the target library according to the parsing result may include the steps of:
1. judging the type of the analysis result;
2. when the analysis result is an insertion operation, an update operation or a deletion operation, writing the corresponding insertion operation, the update operation or the deletion operation into the target library in sequence according to the execution sequence of the transaction for processing the insertion operation, the update operation or the deletion operation;
3. when the analysis result is a commit or rollback operation, writing the commit or rollback operation into the same transaction queue;
4. judging the sequence of the transaction processing the submission or rollback operation in the transaction queue;
5. and writing the corresponding submission or rollback operation into the target library in sequence according to the sequence.
Exemplarily, referring to fig. 2, the source library is explained by taking three transactions (transaction 1, transaction 2, and transaction 3) executed concurrently as an example:
analyzing the log data generated by the source library to obtain an analysis result, wherein the analysis result comprises change operations from three different transactions; judging the type of the change operation:
if the change operation is an insert operation, an update operation or a delete operation, writing the insert operation, the update operation or the delete operation into respective transaction queues (a transaction 1 queue, a transaction 2 queue and a transaction 3 queue), processing the change operation by corresponding transaction threads (a transaction thread 1, a transaction thread 2 and a transaction thread 3), and sequentially writing the corresponding insert operation, the update operation or the delete operation into a target library according to the execution sequence of the transaction 1, the transaction 2 and the transaction 3;
if the change operation is a commit or rollback operation, the commit or rollback operation is written into the respective transaction queue (transaction 1 queue, transaction 2 queue, and transaction 3 queue) while the commit or rollback operation is also written into the same transaction queue (transaction commit/rollback queue), the order of the transaction in the transaction commit/rollback queue for processing the commit or rollback operation is determined, and the corresponding commit or rollback operation is written into the target library according to the order of the transaction.
The above-described approach may ensure that concurrent transactions at the source repository can be restored at the target repository and ensure that the commit/rollback order is consistent with the source repository .
In addition, when the increment synchronous process is abnormal, the scheme can also realize breakpoint continuous transmission, and the specific steps are as follows:
1. monitoring an incremental synchronization process;
2. if abnormal interruption occurs in the increment synchronization, recording and storing the position point when the latest times of increment synchronization is carried out to the target library in real time before the abnormal interruption;
3. and determining the position as an abnormal interruption point, and performing incremental synchronization again according to the abnormal interruption point.
Because breakpoint resume relies on recording and storing breakpoint information, the scheme can realize breakpoint resume in the incremental synchronization process by recording and storing the breakpoint information at the position point during incremental synchronization in real time.
In addition, because the configuration connection from the multiple source libraries to the multiple target libraries is realized, the scheme can also realize the incremental data synchronization from the multiple sub-libraries of the source libraries to the multiple sub-libraries of the target libraries, and the specific implementation process is not repeated.
Example two
As shown in fig. 3, the present application provides DB2 database data synchronization apparatuses, including:
an obtaining module 30, configured to obtain a data synchronization request for a source library to be synchronized and a corresponding target library to be synchronized;
, a synchronization module 31, configured to perform full synchronization on the target repository according to the data of the source repository if the request is a full synchronization request;
and the second synchronization module 32 is configured to, if the request is an incremental synchronization request, obtain and analyze log data generated by the source library in real time, and perform incremental synchronization on the target library according to an analysis result.
Preferably, the above apparatus further comprises:
the configuration module 33 is configured to determine the database partitioning information of the source database and the corresponding database partitioning information of the target database;
and establishing communication connection between the sub-libraries of the source library and the sub-libraries of the target library according to the middleware.
Preferably, the above apparatus further comprises:
the monitoring module 34 is configured to create a cluster management node, and monitor a synchronization process from a library of a source library to a library of the target library in real time according to the cluster management node;
and if the synchronization process from any of the source library to the target library is abnormal, stopping the synchronization process from the rest of the source library to the target library.
Preferably, the second synchronization module 32 includes:
a log analysis module 321, configured to call a jni method at the Java end, and enter a c program;
analyzing data synchronization request parameters transmitted by a Java end in a program c; wherein, the data synchronization request parameter comprises a database site;
determining an initial position of incremental synchronization, and acquiring log data generated after the initial position;
analyzing the log data according to a predefined DB2 log protocol to obtain an analysis result;
and returning the analysis result to the Java end according to the callback Java method.
Preferably, the monitoring module 34 is specifically configured to monitor an incremental synchronization process;
the above-mentioned device still includes:
the recording module 35 is configured to record and store, in real time, a location at which the latest times of increments are synchronized to the target library before the exception interrupt if the increment synchronization has the exception interrupt;
and the abnormal process processing module 36 is configured to determine the position as an abnormal interruption point, and perform incremental synchronization again according to the abnormal interruption point to implement breakpoint resuming in the incremental synchronization process.
Preferably, the second synchronization module 32 further includes:
a concurrent transaction processing module 322, configured to determine a type of the analysis result;
when the analysis result is a commit or rollback operation, writing the commit or rollback operation into the same transaction queue;
judging the sequence of the transaction processing the submission or rollback operation in the transaction queue;
and writing the corresponding submission or rollback operation into the target library in sequence according to the sequence.
Preferably, the monitoring module 34 is further configured to monitor the full-scale synchronization process;
the recording module 35 is further configured to record and store the table name of the source library in real time when performing full-scale synchronization;
the abnormal process processing module 36 is further configured to, if abnormal interruption occurs during full synchronization, obtain an abnormal interruption point according to the table name of the source library recorded and stored in real time, and perform full synchronization again according to the abnormal interruption point to implement table-level breakpoint transmission in the full synchronization process.
Preferably, the above apparatus further comprises:
a sorting module 37, configured to sort every rows of the table of the source library, and obtain primary key values of every rows after sorting;
the recording module 35 is further configured to record and store the primary key value of each rows of the table of the source library in real time when performing full synchronization;
the abnormal process processing module 36 is further configured to, if abnormal interruption occurs during full synchronization, obtain an abnormal interruption point according to the primary key value of each rows of the table of the source library recorded and stored in real time, and perform full synchronization again according to the abnormal interruption point to implement breakpoint continuous transmission at the record level during the full synchronization.
EXAMPLE III
Corresponding to the above method and apparatus, a third embodiment of the present application provides computer systems, including:
or more processors, and
memory associated with the or more processors for storing program instructions that when read and executed by the or more processors perform the following:
acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
if the request is a full-volume synchronization request, performing full-volume synchronization on the target library according to the data of the source library;
and if the request is an incremental synchronization request, acquiring and analyzing the log data generated by the source library in real time, and performing incremental synchronization on the target library according to an analysis result.
FIG. 4 illustrates an architecture of a computer system that may include, in particular, a processor 42, a video display adapter 44, a disk drive 46, an input/output interface 48, a network interface 410, and a memory 412. The processor 42, video display adapter 44, disk drive 46, input/output interface 48, network interface 410, and memory 412 may be communicatively coupled via a communication bus 414.
The processor 42 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or or more Integrated circuits, and is configured to execute related programs to implement the technical solution provided by the present Application.
The Memory 412 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random access Memory), a static storage device, a dynamic storage device, or the like. The memory 412 may store an operating system 416 for controlling the operation of the computer system 40, a Basic Input Output System (BIOS)418 for controlling low-level operations of the computer system. In addition, a web browser 420, a data storage management system 422, and the like may also be stored. In summary, when the technical solution provided by the present application is implemented by software or firmware, the relevant program codes are stored in the memory 412 and called for execution by the processor 42.
The input/output interface 48 is used for connecting an input/output module to realize information input and output. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The network interface 410 is used for connecting a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
The communication bus 414 includes paths that carry information between the various components of the device, such as the processor 42, the video display adapter 44, the disk drive 46, the input/output interface 48, the network interface 410, and the memory 412.
In addition, the computer system can also obtain the information of specific receiving conditions from the virtual resource object receiving condition information database for condition judgment and the like.
It should be noted that although the above-described device only shows the processor 42, the video display adapter 44, the disk drive 46, the input/output interface 48, the network interface 410, the memory 412, the communication bus 414, etc., in a specific implementation, the device may also include other components necessary to achieve proper operation. Furthermore, it will be understood by those skilled in the art that the apparatus described above may also include only the components necessary to implement the solution of the present application, and not necessarily all of the components shown in the figures.
Based on the understanding that the technical solutions of the present application or portions thereof contributing to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing computer devices (which may be personal computers, cloud servers, or network devices, etc.) to execute the methods described in the various embodiments or some portions of the embodiments of the present application.
Although the preferred embodiments of the present invention have been described, those skilled in the art, having the benefit of the present disclosure, may make further changes and modifications to these embodiments without departing from the scope of the present invention, and therefore, it is intended that the appended claims be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1, DB2 database data synchronization method, characterized in that, the method includes:
acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
if the request is a full-scale synchronization request, performing full-scale synchronization on the target library according to the data of the source library;
and if the request is an incremental synchronization request, acquiring and analyzing the log data generated by the source library in real time, and performing incremental synchronization on the target library according to an analysis result.
2. The method of claim 1, further comprising:
determining the database dividing information of the source database and the corresponding database dividing information of the target database;
and establishing communication connection between the sub-libraries of the source library and the sub-libraries of the target library according to the middleware.
3. The method of claim 2, further comprising:
creating a cluster management node, and monitoring the synchronous process from the sub-library of the source library to the sub-library of the target library in real time according to the cluster management node;
and if the synchronization process from any of the source library to the target library is abnormal, stopping the synchronization process from the rest of the source library to the target library.
4. The method according to claim 1, wherein the obtaining and analyzing the log data generated by the source library in real time specifically comprises:
calling a jni method at a Java end, and entering a program c;
analyzing data synchronization request parameters transmitted by the Java terminal in the c program; wherein the data synchronization request parameter comprises a database location;
determining an initial position of incremental synchronization, and acquiring the log data generated after the initial position;
analyzing the log data according to a predefined DB2 log protocol to obtain an analysis result;
and returning the analysis result to the Java end according to the callback Java method.
5. The method of claim 4, further comprising:
monitoring the incremental synchronization process;
if abnormal interruption occurs in the increment synchronization, recording and storing the position of the latest times of increment synchronization to the target library in real time before the abnormal interruption;
and determining the position as an abnormal interruption point, and performing incremental synchronization again according to the abnormal interruption point to realize breakpoint continuous transmission in the incremental synchronization process.
6. The method according to claim 1, wherein the performing incremental synchronization on the target library to be synchronized according to the parsing result specifically comprises:
judging the type of the analysis result;
when the analysis result is a commit or rollback operation, writing the commit or rollback operation into a transaction queue of the same ;
judging the sequence of the transaction for processing the commit or rollback operation in the transaction queue;
and writing the corresponding submission or rollback operation into the target library in sequence according to the sequence.
7. The method of claim 1, further comprising:
monitoring the full-scale synchronization process;
recording and storing the table name of the source library in real time when full synchronization is carried out;
and if abnormal interruption occurs in the full synchronization, positioning according to the table name of the source library recorded and stored in real time to obtain an abnormal interruption point, and performing the full synchronization again according to the abnormal interruption point to realize the breakpoint continuous transmission of the table level in the full synchronization process.
8. The method of claim 7, further comprising:
sequencing every rows of the table of the source library, and acquiring primary key values of every rows after sequencing;
recording and storing primary key values of every rows of the table of the source library in real time when the full-scale synchronization is carried out;
and if the full synchronization has abnormal interruption, positioning according to primary key values of every rows of the table of the source library recorded and stored in real time to obtain an abnormal interruption point, and performing the full synchronization again according to the abnormal interruption point to realize the breakpoint continuous transmission of the record level in the full synchronization process.
9, DB2 database data synchronization device, characterized in that, the device includes:
the acquisition module is used for acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
, a synchronization module, configured to perform full synchronization on the target repository according to the data of the source repository if the request is a full synchronization request;
and the second synchronization module is used for acquiring and analyzing the log data generated by the source library in real time if the request is an incremental synchronization request, and performing incremental synchronization on the target library according to an analysis result.
10, , a computer system, comprising:
or more processors, and
memory associated with the or more processors for storing program instructions that when read executed by the or more processors perform the following:
acquiring data synchronization requests aiming at a source library to be synchronized and a corresponding target library to be synchronized;
if the request is a full-scale synchronization request, performing full-scale synchronization on the target library according to the data of the source library;
and if the request is an incremental synchronization request, acquiring and analyzing the log data generated by the source library in real time, and performing incremental synchronization on the target library according to an analysis result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910842876.3A CN110737720A (en) | 2019-09-06 | 2019-09-06 | DB2 database data synchronization method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910842876.3A CN110737720A (en) | 2019-09-06 | 2019-09-06 | DB2 database data synchronization method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110737720A true CN110737720A (en) | 2020-01-31 |
Family
ID=69267487
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910842876.3A Pending CN110737720A (en) | 2019-09-06 | 2019-09-06 | DB2 database data synchronization method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737720A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339103A (en) * | 2020-03-13 | 2020-06-26 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full fragmentation and incremental log analysis |
| CN112667698A (en) * | 2021-01-04 | 2021-04-16 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
| CN112948494A (en) * | 2021-03-04 | 2021-06-11 | 北京沃东天骏信息技术有限公司 | Data synchronization method and device, electronic equipment and computer readable medium |
| CN113297326A (en) * | 2021-05-21 | 2021-08-24 | 中国邮政储蓄银行股份有限公司 | Data processing method and device, computer readable storage medium and processor |
| CN113918657A (en) * | 2021-12-14 | 2022-01-11 | 天津南大通用数据技术股份有限公司 | Parallel high-performance incremental synchronization method |
| CN114003622A (en) * | 2021-12-30 | 2022-02-01 | 天津南大通用数据技术股份有限公司 | Huge transaction increment synchronization method between transaction type databases |
| CN115599870A (en) * | 2022-12-15 | 2023-01-13 | 云筑信息科技（成都）有限公司（Cn） | Data synchronization method based on fusion of stock data and incremental data of message queue |
| CN117171266A (en) * | 2023-08-28 | 2023-12-05 | 北京逐风科技有限公司 | Data synchronization method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102495906A (en) * | 2011-12-23 | 2012-06-13 | 天津神舟通用数据技术有限公司 | Incremental data migration method capable of realizing breakpoint transmission |
| CN105243067A (en) * | 2014-07-07 | 2016-01-13 | 北京明略软件系统有限公司 | Method and apparatus for realizing real-time increment synchronization of data |
| CN107659663A (en) * | 2017-10-27 | 2018-02-02 | 北京锐安科技有限公司 | A kind of method, apparatus of data syn-chronization, equipment and storage medium |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| US20180341560A1 (en) * | 2017-05-23 | 2018-11-29 | International Business Machines Corporation | Service outage time reduction for a planned event in a system |
-
2019
- 2019-09-06 CN CN201910842876.3A patent/CN110737720A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102495906A (en) * | 2011-12-23 | 2012-06-13 | 天津神舟通用数据技术有限公司 | Incremental data migration method capable of realizing breakpoint transmission |
| CN105243067A (en) * | 2014-07-07 | 2016-01-13 | 北京明略软件系统有限公司 | Method and apparatus for realizing real-time increment synchronization of data |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| US20180341560A1 (en) * | 2017-05-23 | 2018-11-29 | International Business Machines Corporation | Service outage time reduction for a planned event in a system |
| CN107659663A (en) * | 2017-10-27 | 2018-02-02 | 北京锐安科技有限公司 | A kind of method, apparatus of data syn-chronization, equipment and storage medium |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339103A (en) * | 2020-03-13 | 2020-06-26 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full fragmentation and incremental log analysis |
| CN111339103B (en) * | 2020-03-13 | 2023-06-20 | 河南安冉云网络科技有限公司 | Data exchange method and system based on full-quantity fragmentation and incremental log analysis |
| CN112667698A (en) * | 2021-01-04 | 2021-04-16 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
| CN112667698B (en) * | 2021-01-04 | 2023-02-28 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
| CN112948494A (en) * | 2021-03-04 | 2021-06-11 | 北京沃东天骏信息技术有限公司 | Data synchronization method and device, electronic equipment and computer readable medium |
| CN113297326A (en) * | 2021-05-21 | 2021-08-24 | 中国邮政储蓄银行股份有限公司 | Data processing method and device, computer readable storage medium and processor |
| CN113918657B (en) * | 2021-12-14 | 2022-03-15 | 天津南大通用数据技术股份有限公司 | Parallel high-performance incremental synchronization method |
| CN113918657A (en) * | 2021-12-14 | 2022-01-11 | 天津南大通用数据技术股份有限公司 | Parallel high-performance incremental synchronization method |
| CN114003622A (en) * | 2021-12-30 | 2022-02-01 | 天津南大通用数据技术股份有限公司 | Huge transaction increment synchronization method between transaction type databases |
| CN115599870A (en) * | 2022-12-15 | 2023-01-13 | 云筑信息科技（成都）有限公司（Cn） | Data synchronization method based on fusion of stock data and incremental data of message queue |
| CN115599870B (en) * | 2022-12-15 | 2023-04-07 | 云筑信息科技(成都)有限公司 | Data synchronization method based on fusion of stock data and incremental data of message queue |
| CN117171266A (en) * | 2023-08-28 | 2023-12-05 | 北京逐风科技有限公司 | Data synchronization method, device, equipment and storage medium |
| CN117171266B (en) * | 2023-08-28 | 2024-05-14 | 北京逐风科技有限公司 | Data synchronization method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737720A (en) | DB2 database data synchronization method, device and system | |
| CN109144994B (en) | Index updating method, system and related device | |
| CN110147407B (en) | Data processing method and device and database management server | |
| US20180081956A1 (en) | Method for automatically synchronizing multi-source heterogeneous data resources | |
| CN111324610A (en) | Data synchronization method and device | |
| US11928089B2 (en) | Data processing method and device for distributed database, storage medium, and electronic device | |
| CN108205560B (en) | Data synchronization method and device | |
| US11526475B2 (en) | Code generator platform for data transformation | |
| CN106648994B (en) | Method, equipment and system for backing up operation log | |
| CN110737594A (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN113722277A (en) | Data import method, device, service platform and storage medium | |
| CN108363787B (en) | IFC file parallel storage method, device and system | |
| CN114416868A (en) | Data synchronization method, device, equipment and storage medium | |
| CN115237889A (en) | Database switching method and device, storage medium and computer equipment | |
| CN115113989B (en) | Transaction execution method, device, computing equipment and storage medium | |
| CN115080666A (en) | Data synchronization method, system, electronic device and storage medium | |
| CN114840562A (en) | Distributed caching method and device for business data, electronic equipment and storage medium | |
| CN114547206A (en) | Data synchronization method and data synchronization system | |
| WO2022159391A1 (en) | Handling system-characteristics drift in machine learning applications | |
| CN110168514B (en) | Transaction processing method, device and equipment | |
| CN112699129A (en) | Data processing system, method and device | |
| CN113779117A (en) | Data monitoring method and device, storage medium and electronic equipment | |
| CN112000671A (en) | Block chain-based database table processing method, device and system | |
| US12032533B2 (en) | Code generator platform for data transformation | |
| US9818078B1 (en) | Converting a non-workflow program to a workflow program using workflow inferencing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200131 |
|
| RJ01 | Rejection of invention patent application after publication |