CN111382201A - Heterogeneous database synchronization method and device, computer equipment and storage medium - Google Patents
Heterogeneous database synchronization method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111382201A CN111382201A CN201811644117.8A CN201811644117A CN111382201A CN 111382201 A CN111382201 A CN 111382201A CN 201811644117 A CN201811644117 A CN 201811644117A CN 111382201 A CN111382201 A CN 111382201A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- target transmission
- heterogeneous database
- heterogeneous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a heterogeneous database synchronization method, a heterogeneous database synchronization device, computer equipment and a storage medium, wherein the heterogeneous database synchronization method comprises the following steps: acquiring incremental data of a database; packaging the incremental data of the database into target transmission data according to a preset custom packaging protocol; and transmitting the target transmission data into the heterogeneous database system so that the target transmission data is synchronously transmitted in a plurality of databases in the heterogeneous database system. According to the embodiment of the invention, the incremental data of the database are acquired, and then the incremental data of the database are packaged into the target transmission data according to the preset custom packaging protocol and then transmitted to the heterogeneous database system, so that the target transmission data are synchronously transmitted by a plurality of databases in the heterogeneous database system, the data collision possibly occurring in the data transmission process can be effectively avoided, the final consistency of the data is realized, the iteration performance of the database version at the bottom layer is improved, and the replacement and the migration of the database are convenient.
Description
Technical Field
The embodiment of the invention relates to the technical field of data synchronization, in particular to a heterogeneous database synchronization method, a heterogeneous database synchronization device, computer equipment and a storage medium.
Background
With the development of network technology, databases tend to develop in a high-availability direction more and more, in some existing schemes, a data conflict solution algorithm can be uniformly controlled by setting proxy layer hijack service access, namely, the data conflict solution algorithm is quickly written in the local by invading an original database, but the scheme has no universality and does not support heterogeneous databases. In other schemes, a heterogeneous database can be adopted, the heterogeneous database can realize data sharing and transparent access, a plurality of database systems exist before being added into the heterogeneous database system, each component part which has the own data array management system and the external database has autonomy, and each database system still has own application characteristics, integrity control and safety control while realizing data sharing. Existing databases typically employ a one-master-multiple-slave approach, where a write operation acts on a master and a read operation acts on a slave, the master acts as the only writable point, and data synchronization is unidirectional.
However, for the one-master multi-slave mode, all write operations are completed at the master point, so that the time consumption greatly depends on the network condition between the service point and the master point, for example, in the process of global service deployment, when the service point is in europe and the data point is in asia, the data transmission between the service point and the data point is affected by the network quality, and the problem of long time consumption of data transmission is caused.
Disclosure of Invention
The embodiment of the invention provides a heterogeneous database synchronization method, a heterogeneous database synchronization device, computer equipment and a storage medium, wherein incremental data are encapsulated through a custom encapsulation protocol to achieve mutual data synchronization between heterogeneous databases.
In order to solve the above technical problem, the embodiment of the present invention adopts a technical solution that: the heterogeneous database synchronization method comprises the following steps:
acquiring incremental data of a database;
packaging the incremental data of the database into target transmission data according to a preset custom packaging protocol;
and transmitting the target transmission data to a heterogeneous database system so that the target transmission data is synchronously transmitted in a plurality of databases in the heterogeneous database system.
Optionally, the step of encapsulating the database incremental data into target transmission data according to a preset custom encapsulation protocol includes the following steps:
acquiring local interface information and mounting a preset packaging plug-in according to the interface information;
and analyzing and packaging the incremental data of the database into target transmission data in a unified log format through the packaging plug-in.
Optionally, the step of transmitting the target transmission data to a heterogeneous database system, so that the target transmission data is transmitted in several databases in the heterogeneous database system synchronously includes the following steps:
acquiring data conflict information of the plurality of databases for synchronous transmission of the target transmission data;
and analyzing the data conflict information according to a preset conflict resolution strategy to generate conflict resolution information, so that the databases execute conflict resolution operation according to the conflict resolution information to perform synchronous transmission.
Optionally, the step of obtaining data collision information for performing synchronous transmission of the target transmission data by the databases includes the following steps:
establishing a unique version identifier for the incremental record generated by synchronously transmitting the plurality of databases;
and calculating the unique version identification according to a preset judgment rule to generate the data conflict information.
Optionally, after the step of transmitting the target transmission data to the heterogeneous database system where the target database is located, so that the target transmission data is synchronously transmitted in a plurality of databases in the heterogeneous database system, the method further includes the following steps:
acquiring preset local data format information;
and integrating the data formats of all databases in the heterogeneous database system into a data format matched with the data format information according to the local data format information.
Optionally, after the step of integrating the data formats of the databases in the heterogeneous database system into the data format matched with the data format information according to the local data format information, the method further includes the following steps:
acquiring change information of data in each database in the heterogeneous database system;
and generating a change data table of each database according to the change information.
Optionally, the step of transmitting the target transmission data to a heterogeneous database system, so that the target transmission data is synchronously transmitted in several databases in the heterogeneous database system, further includes the following steps:
acquiring the database increment data in the target transmission data and a change record version corresponding to the database increment data;
comparing the version of the change record with the version of the local record;
and broadcasting the target transmission data when the change record version is matched with the local record version, so that a first target database of which the record version is not matched with the change record version in the heterogeneous database system carries out preset updating operation according to the target transmission data.
To solve the foregoing technical problem, an embodiment of the present invention further provides a heterogeneous database synchronization apparatus, including:
the first acquisition module is used for acquiring incremental data of the database;
the first processing module is used for packaging the database incremental data into target transmission data according to a preset custom packaging protocol;
the first execution module is used for transmitting the target transmission data to a heterogeneous database system so as to enable the target transmission data to be synchronously transmitted in a plurality of databases in the heterogeneous database system.
In order to solve the above technical problem, an embodiment of the present invention further provides a computer device, including a memory and a processor, where the memory stores computer-readable instructions, and the computer-readable instructions, when executed by the processor, cause the processor to perform the steps of the above heterogeneous database synchronization method.
To solve the above technical problem, an embodiment of the present invention further provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the heterogeneous database synchronization method.
The embodiment of the invention has the beneficial effects that: by acquiring the incremental data of the databases, packaging the incremental data of the databases into target transmission data according to a preset custom packaging protocol, and transmitting the target transmission data to the heterogeneous database system, a plurality of databases in the heterogeneous database system synchronously transmit the target transmission data, and each database in the heterogeneous database system is used as a main point to execute write operation, so that data collision possibly occurring in the data transmission process can be effectively avoided, the final consistency of the data is realized, the iteration performance of a bottom database version is improved, and the replacement and migration of the databases are facilitated.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic diagram illustrating a basic flow of a heterogeneous database synchronization method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating packaging incremental data of a database according to an embodiment of the present invention;
FIG. 3 is a flow chart illustrating data synchronization of a heterogeneous database system according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a process of generating data collision information according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating a unified data format according to an embodiment of the present invention;
FIG. 6 is a flow chart illustrating a process of recording data changes in a database according to an embodiment of the present invention;
FIG. 7 is a flow chart illustrating database data synchronization according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of a basic structure of a heterogeneous database synchronization apparatus according to an embodiment of the present invention;
FIG. 9 is a block diagram of the basic structure of a computer device according to an embodiment of the present invention;
fig. 10 is a schematic flow chart of a framework for data synchronization in a heterogeneous database system according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention.
In some of the flows described in the present specification and claims and in the above figures, a number of operations are included that occur in a particular order, but it should be clearly understood that these operations may be performed out of order or in parallel as they occur herein, with the order of the operations being indicated as 101, 102, etc. merely to distinguish between the various operations, and the order of the operations by themselves does not represent any order of performance. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
Referring to fig. 1, fig. 1 is a schematic view illustrating a basic flow of the heterogeneous database synchronization method according to the present embodiment.
As shown in fig. 1, a heterogeneous database synchronization method includes the following steps:
s1100, obtaining incremental data of a database;
the database incremental data refers to information data of the database with changed storage data, including but not limited to adding new data, deleting data and modifying data, and when data operation is performed in the database, corresponding log records are generated for recording changes of the data in the database.
S1200, packaging the incremental data of the database into target transmission data according to a preset custom packaging protocol;
after the database incremental data is obtained, the database incremental data may be encapsulated into the target transmission data through a custom encapsulation protocol, where the custom encapsulation protocol is preset by the system for encapsulating data, and specifically, the custom encapsulation protocol may encrypt the database incremental data to generate the target transmission data, for example: the target transmission data is generated by encrypting the database incremental data through a BASE64, RSA or MD5 encryption algorithm, and of course, the operation of changing the data through a custom database may also be performed, for example, the operations of adding, deleting and modifying the data of the database are respectively defined as a1, B1 and C1, and then the specific data of changing the data are merged into the custom operation code, for example, the new data added into the database is "123", and then the database incremental data may be packaged into a1-123, and of course, the working principle of the custom packaging protocol may also adopt other packaging methods to package the database incremental data into the target transmission data. And when the implementation is carried out, the incremental data of the database can be analyzed by mounting a preset plug-in the database, and the incremental data of the database is packaged into target transmission data with a uniform format.
S1300, transmitting the target transmission data to a heterogeneous database system so that the target transmission data is synchronously transmitted in a plurality of databases in the heterogeneous database system.
After the database increment data is encapsulated into target transmission data, the target transmission data is sent to the heterogeneous database system, so that a plurality of (for example, 3, 5 or 10) databases in the heterogeneous database system synchronize the target transmission data, and mutual synchronization of data among the plurality of databases of the heterogeneous database is realized. In operation, please refer to fig. 10, fig. 10 is a block diagram illustrating a framework flow of data synchronization in a heterogeneous database system according to an embodiment of the present invention.
As shown in fig. 10, taking 3 databases included in a heterogeneous database system as an example, the databases are respectively a database a, a database B, and a database C, where the database a, the database B, and the database C are respectively mounted with different plug-ins a and plug-ins B, and the heterogeneous database synchronization method of the present invention is applied to each database of the heterogeneous database system, and is specifically represented as that the database a, the database B, and the database C respectively correspond to an application a, an application B, and an application C, and in implementation, the application a, the application B, and the application C respectively analyze database incremental data generated by the database a, the database B, and the database C and encapsulate the database incremental data into a unified log of the application, which is transmitted between different databases, where the unified log of the application is target transmission data, for example: when the data of the database A is changed, for example, a piece of data D1 is added, the application program A acquires the data D1 through the plug-in A, namely the database incremental data of the database A, then the database incremental data is packaged into target transmission data according to a preset custom packaging protocol and transmitted to the heterogeneous database system, the application program B and the application program C acquire the target transmission data and synchronize to the database B and the database C through the plug-in B and the plug-in C respectively after analyzing, mutual synchronization of data among a plurality of points of heterogeneous data is realized, and final data consistency is realized. It should be noted that the number of databases in the heterogeneous database system is not limited to the above 3, and the number of databases in the heterogeneous database system may also be designed to other values according to different application scenarios.
In the process of data synchronization, a situation of data conflict may occur, at this time, a version record table may be established in the database to record data change records of the database, each change record is provided with a unique version, for one database, when incremental data is synchronously written, a unique version is generated for the incremental data, and the version is judged, the data can be updated if the unique version is higher than the version in the version record table or if no version (i.e. incremental data is written for the first time) is in the version record table, and if the unique version is smaller than the version in the version record table, no operation is performed; and generating a version when incremental data are synchronously transmitted, judging the version, entering a conflict resolution algorithm process if the versions are the same and possibly causing data conflict, generating a unique version for the incremental record through a preset uniform version generation algorithm, and updating the version in a version record table if the versions are different. On the other hand, the database in the heterogeneous database system cannot distinguish whether the business writing or the synchronous (program) writing is carried out, and a loop is caused when the business writing or the synchronous (program) writing is sent out, at the moment, a counteracting version record table can be established in the database, a main key of the counteracting version record table is the only version of the data change record, and the counteracting version record table is written when the incremental data of the database is synchronously written; and when the offset version record table has no record, the corresponding record of the offset version record table is eliminated, and when the offset version record table has no record, the incremental data of the database is transmitted.
In the embodiment, the database incremental data is acquired, then the database incremental data is encapsulated into the target transmission data according to the preset custom encapsulation protocol, and then the target transmission data is transmitted to the heterogeneous database system, so that the target transmission data is synchronously transmitted by a plurality of databases in the heterogeneous database system, and each database in the heterogeneous database system is used as a main point to execute write operation, thereby effectively avoiding data collision possibly occurring in the data transmission process, realizing final data consistency, improving the iteration performance of a bottom database version, and facilitating the replacement and migration of the database.
In an alternative embodiment, please refer to fig. 2, fig. 2 is a flowchart illustrating a process of packaging incremental data of a database according to an embodiment of the present invention.
As shown in fig. 2, step S1200 includes the following steps:
s1210, obtaining local interface information and mounting a preset packaging plug-in according to the interface information;
the method comprises the steps that the incremental data of the database can be packaged through a mounting plug-in, local interface information is obtained, then a preset packaging plug-in is mounted according to the interface information, the interface information refers to a method that the database is provided for the outside for separating external communication from internal operation, the packaging plug-in can be associated and communicated with the database through the interface information, and when the method is implemented, the packaging plug-in is used for analyzing the incremental data of the database and packaging the incremental data of the database according to a preset custom packaging protocol to generate target transmission data.
And S1220, analyzing and packaging the incremental data of the database into target transmission data in a unified log format through the packaging plug-in.
After the packaging plug-in is mounted, the interface analyzes the positive energy data of the database through the packaging plug-in and packages the positive energy data into target transmission data in a unified log format.
In another alternative embodiment, please refer to fig. 3, fig. 3 is a basic flowchart illustrating data synchronization of a heterogeneous database system according to an embodiment of the present invention.
As shown in fig. 3, step S1300 includes the following steps:
s1310, acquiring data conflict information of the plurality of databases for synchronous transmission of the target transmission data;
when each database in the heterogeneous database system carries out data synchronization, because each database is a main point and can carry out write operation, data collision can occur when the data synchronization is not unidirectional, by acquiring data collision information generated by synchronous transmission of target transmission data of each database in the heterogeneous database system, taking a database a change data as an example, the system packages database incremental data of the database a change data into target transmission data and transmits the target transmission data to the heterogeneous database system, a database b and a database c in the heterogeneous database are both connected with the database a, data synchronization is realized after the target transmission data are received, a custom packaging protocol is carried out on the data change of the heterogeneous database b and the database c while the data are synchronized to form the target transmission data to be transmitted to the heterogeneous database system, and a database d in the heterogeneous database system can receive the target transmission data of the database a, For example, the target transmission data sent by the database b and the database c is received first and data change is performed according to the target transmission data to realize data synchronization, and when the target transmission data sent by the database b and the database c is received, data synchronization is already completed by the database c, and data collision information is generated at this time.
S1320, analyzing the data conflict information according to a preset conflict resolution strategy to generate conflict resolution information, so that the databases execute conflict resolution operation according to the conflict resolution information to perform synchronous transmission.
After the data conflict information is acquired, analyzing the data conflict information according to a preset conflict resolution strategy to generate conflict resolution information, so that a database can execute conflict resolution operation according to the conflict resolution information, wherein the working principle of the conflict resolution strategy is as follows: analyzing the data conflict information, for example, when the database c synchronizes the incremental data d, a data conflict occurs, so as to generate conflict solution information that the database c performs a database data change operation on the incremental data d, so that the database c performs the conflict solution operation on the incremental data d according to the conflict solution information, for example, the database c does not perform the data change operation on the incremental data d and records the data change operation, thereby preventing the database c from performing repeated operations on the incremental data d and solving the data conflict.
In an alternative embodiment, please refer to fig. 4, where fig. 4 is a basic flowchart illustrating the generation of data collision information according to an embodiment of the present invention.
As shown in fig. 4, step S1310 includes the following steps:
s1311, establishing a unique version identifier for the incremental record generated by synchronous transmission of the databases;
when the databases in the heterogeneous database system are synchronized, a unique version identifier can be established for an incremental record generated by synchronous transmission of each database in the heterogeneous database system, that is, a unique version identifier is generated every time data of the database is changed.
S1312, calculating the unique version identification according to a preset judgment rule to generate the data collision information.
After the unique version identification is established for the incremental record, the unique version identification is calculated according to a preset judgment rule to generate data conflict information, taking the case of writing incremental data into the database, because each data change of the database has a unique version identification, therefore, before the incremental data is written, the unique version identification of the written incremental data is compared with the unique version identification of each data change in the database, if the unique version identification of the written incremental data is larger than the unique version identification of each data change in the database, the incremental data is written and the unique version identification of the database is updated, when the unique version identification of the written incremental data is equal to or less than the unique version identification of each data change in the database, the database has already performed the operation of the incremental data, and data collision information is generated, including the identifier of the database and the incremental data corresponding to the occurrence of the data collision.
In an alternative embodiment, please refer to fig. 5, fig. 5 is a basic flowchart illustrating a unified data format according to an embodiment of the present invention.
As shown in fig. 5, after step S1300, the following steps are further included:
s1400, acquiring preset local data format information;
the local data format information refers to information which is preset by the system and used for controlling the data format stored in the database, when the system is implemented, the system is provided with a format data table, the format data table stores the local data format information, and the system can acquire the local data format information according to the root data table.
S1500, integrating the data formats of all databases in the heterogeneous database system into a data format matched with the data format information according to the local data format information.
After the local data format information is acquired, the data formats of all databases in the heterogeneous database system are integrated into a uniform data format according to the local data format information, and the integrity of data in the heterogeneous database system is ensured by integrating the data formats into the uniform data format.
In an alternative embodiment, please refer to fig. 6, fig. 6 is a schematic diagram of a basic flow of recording data changes of a database according to an embodiment of the present invention.
As shown in fig. 6, step S1500 is followed by the following steps:
s1600, acquiring the change information of the data in each database in the heterogeneous database system;
and after the databases in the heterogeneous database system finish data synchronization, acquiring the change information of the data of each database for data change. In implementation, the database is provided with a data change record table, when data change occurs in the database, for example, addition, deletion and modification data are all recorded in the data change record table, and the system can acquire change information of data change of each database in the heterogeneous database system according to the data change record table.
And S1700, generating a change data table of each database according to the change information.
After the change information of the data change of the database is acquired, the change data table of each data is generated according to the change information, so that the statistics of the data change of each database in the heterogeneous database is realized, and the change or operation of the data in the database at the later stage is conveniently checked and analyzed.
In an alternative embodiment, please refer to fig. 7, fig. 7 is a basic flowchart of database data synchronization according to an embodiment of the present invention.
As shown in fig. 7, step S1300 includes the following steps:
s1330, obtaining the database incremental data in the target transmission data and a change record version corresponding to the database incremental data;
after a database in the heterogeneous database system obtains target transmission data, analyzing the target transmission data according to a custom packaging protocol to obtain database incremental data in the target transmission data and a change record version corresponding to the database incremental data, wherein the change record version is generated for the database incremental data by a database in which the database incremental data occurs, taking the case that the database a has data change as an example, the database a sets a unique change record version for the data change, namely, the database incremental data sets a corresponding change record version.
S1340, comparing the version of the change record with the version of the local record;
after obtaining the change record version of the incremental data of the database, comparing the change record version with the local record version, wherein the local record version refers to the history record of the data change of the database, for example, a change record is generated when adding data to the database, another change record is generated when deleting data, and another change record is generated when modifying data, that is, a change record is generated when data change occurs in the database, and the change record is provided with a unique version, taking the heterogeneous database synchronization method of the present invention applied to the database b as an example, for the target transmission data transmitted from the database a to the heterogeneous database system, the target transmission data comprises the incremental data of the data change of the database a and the change record version corresponding to the incremental data, and the database b obtains the incremental data and the change record version of the target transmission data, and comparing the version of the change record with the version of the local record so as to judge whether the operation of the incremental data is executed.
And S1350, broadcasting the target transmission data when the change record version is matched with the local record version, so that a first target database in the heterogeneous database system, of which the record version is not matched with the change record version, performs preset updating operation according to the target transmission data.
When the change record version is matched with the local record version, namely the database completes data synchronization on the incremental data, at the moment, the target transmission data is broadcasted, so that the database in the heterogeneous database system performs preset updating operation on the target transmission database, the updating operation comprises that when the incremental data is synchronously written, the local record version and the change record version are judged, if the change record version is higher than the local record version, the data updating is performed on the incremental data, if the change record version is lower than the local record version, the incremental data is not operated, and of course, if the change record version is not changed, the incremental data is operated for the first time, and the incremental data is updated; on the other hand, when the incremental data are synchronously transmitted, a change record version of the incremental data is generated and compared with a local record version, if the versions are matched, a unique change record version is generated for the record which transmits the sorted data and is compared with the local record version, and when the versions are not matched, the incremental data are transmitted and the local record version is updated. In a heterogeneous database system, as a database cannot distinguish whether business writing or synchronous writing is carried out and is sent out, the problem of data loop is caused, a counteracting version table can be established at the moment, a main key of the counteracting version table is the only version of a change record, and when incremental data is synchronously written, the counteracting version table is written at the same time, namely the only version of the incremental data written in the counteracting version table is updated; when incremental data are synchronously transmitted, if records exist in the offset version table, the incremental data are not transmitted, meanwhile, the corresponding records of the offset version table are eliminated, and if no records exist in the offset version table, the incremental data are transmitted, so that the problem of loop back can be effectively avoided, and the final consistency of the data is realized.
In order to solve the above technical problem, an embodiment of the present invention further provides a heterogeneous database synchronization apparatus.
Referring to fig. 8, fig. 8 is a schematic diagram illustrating a basic structure of the synchronization apparatus for heterogeneous databases according to the present embodiment.
As shown in fig. 8, a heterogeneous database synchronization apparatus includes: a first obtaining module 2100, a first processing module 2200 and a first executing module 2300, wherein the first obtaining module 2100 is configured to obtain database incremental data; the first processing module 2200 is configured to encapsulate the database incremental data into target transmission data according to a preset custom encapsulation protocol; the first execution module 2300 is configured to transmit the target transmission data to heterogeneous database systems, so that the target transmission data is transmitted in several databases in the heterogeneous database systems synchronously.
In the embodiment, the database incremental data is acquired, then the database incremental data is encapsulated into the target transmission data according to the preset custom encapsulation protocol, and then the target transmission data is transmitted to the heterogeneous database system, so that the target transmission data is synchronously transmitted by a plurality of databases in the heterogeneous database system, and each database in the heterogeneous database system is used as a main point to execute write operation, thereby effectively avoiding data collision possibly occurring in the data transmission process, realizing final data consistency, improving the iteration performance of a bottom database version, and facilitating the replacement and migration of the database.
In some embodiments, the heterogeneous database synchronization apparatus further comprises: the system comprises a first obtaining submodule and a first executing submodule, wherein the first obtaining submodule is used for obtaining local interface information and mounting a preset packaging plug-in according to the interface information; the first execution submodule is used for analyzing and packaging the database incremental data into target transmission data in a unified log format through the packaging plug-in.
In some embodiments, the heterogeneous database synchronization apparatus further comprises: the second acquisition submodule is used for acquiring data conflict information of the plurality of databases for synchronous transmission of the target transmission data; and the second execution sub-module is used for analyzing the data conflict information according to a preset conflict resolution strategy to generate conflict resolution information, so that the databases execute conflict resolution operation according to the conflict resolution information to perform synchronous transmission.
In some embodiments, the heterogeneous database synchronization apparatus further comprises: the first processing submodule is used for establishing a unique version identifier for incremental records generated by synchronous transmission of the databases; and the third execution submodule is used for calculating the unique version identification according to a preset judgment rule to generate the data conflict information.
In some embodiments, the heterogeneous database synchronization apparatus further comprises: the second acquisition module is used for acquiring preset local data format information; and the second execution module is used for integrating the data formats of all databases in the heterogeneous database system into a data format matched with the data format information according to the local data format information.
In some embodiments, the heterogeneous database synchronization apparatus further comprises: the heterogeneous database system comprises a third acquisition module and a third execution module, wherein the third acquisition module is used for acquiring the change information of data in each database in the heterogeneous database system; and the third execution module is used for generating a change data table of each database according to the change information.
In some embodiments, the heterogeneous database synchronization apparatus further comprises: the third obtaining submodule is used for obtaining the database incremental data in the target transmission data and a change record version corresponding to the database incremental data; the comparison submodule is used for comparing the version of the change record with the version of the local record; and the fourth execution sub-module is used for broadcasting the target transmission data when the change record version is matched with the local record version, so that a first target database of which the record version is not matched with the change record version in the heterogeneous database system performs preset updating operation according to the target transmission data.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
In order to solve the above technical problem, an embodiment of the present invention further provides a computer device. Referring to fig. 9, fig. 9 is a block diagram of a basic structure of a computer device according to the present embodiment.
As shown in fig. 9, the internal structure of the computer device is schematically illustrated. As shown in fig. 9, the computer apparatus includes a processor, a nonvolatile storage medium, a memory, and a network interface connected through a system bus. The non-volatile storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store control information sequences, and the computer readable instructions when executed by the processor can enable the processor to realize a heterogeneous database synchronization method. The processor of the computer device is used for providing calculation and control capability and supporting the operation of the whole computer device. The memory of the computer device may have stored therein computer readable instructions that, when executed by the processor, may cause the processor to perform a heterogeneous database synchronization method. The network interface of the computer device is used for connecting and communicating with the terminal. It will be appreciated by those skilled in the art that the configurations shown in the figures are block diagrams of only some of the configurations relevant to the present application, and do not constitute a limitation on the computing devices to which the present application may be applied, and that a particular computing device may include more or less components than those shown in the figures, or may combine certain components, or have a different arrangement of components.
In this embodiment, the processor is configured to execute the first obtaining module 2100, the first processing module 2200, and the first executing module 2300 in fig. 8, and the memory stores program codes and various data required for executing the modules. The network interface is used for data transmission to and from a user terminal or a server. The memory in this embodiment stores program codes and data required for executing all the sub-modules in the heterogeneous database synchronization device, and the server can call the program codes and data of the server to execute the functions of all the sub-modules.
The computer acquires the database incremental data, packages the database incremental data into target transmission data according to a preset custom packaging protocol, and transmits the target transmission data to the heterogeneous database system, so that a plurality of databases in the heterogeneous database system synchronously transmit the target transmission data, and each database in the heterogeneous database system is used as a main point to execute write operation, thereby effectively avoiding data collision possibly occurring in the data transmission process, realizing final data consistency, improving the iteration performance of a bottom-layer database version, and facilitating the replacement and migration of the databases.
The present invention also provides a storage medium storing computer-readable instructions, which when executed by one or more processors, cause the one or more processors to perform the steps of the heterogeneous database synchronization method according to any one of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A heterogeneous database synchronization method is characterized by comprising the following steps:
acquiring incremental data of a database;
packaging the incremental data of the database into target transmission data according to a preset custom packaging protocol;
and transmitting the target transmission data to a heterogeneous database system so that the target transmission data is synchronously transmitted in a plurality of databases in the heterogeneous database system.
2. The heterogeneous database synchronization method according to claim 1, wherein the step of encapsulating the database incremental data into target transmission data according to a preset custom encapsulation protocol comprises the steps of:
acquiring local interface information and mounting a preset packaging plug-in according to the interface information;
and analyzing and packaging the incremental data of the database into target transmission data in a unified log format through the packaging plug-in.
3. The heterogeneous database synchronization method according to claim 1, wherein the step of transmitting the target transmission data to a heterogeneous database system so that the target transmission data is transmitted synchronously in several databases in the heterogeneous database system comprises the following steps:
acquiring data conflict information of the plurality of databases for synchronous transmission of the target transmission data;
and analyzing the data conflict information according to a preset conflict resolution strategy to generate conflict resolution information, so that the databases execute conflict resolution operation according to the conflict resolution information to perform synchronous transmission.
4. The heterogeneous database synchronization method according to claim 3, wherein the step of obtaining data collision information of the databases for synchronous transmission of the target transmission data comprises the steps of:
establishing a unique version identifier for the incremental record generated by synchronously transmitting the plurality of databases;
and calculating the unique version identification according to a preset judgment rule to generate the data conflict information.
5. The heterogeneous database synchronization method according to claim 1, wherein after the step of transmitting the target transmission data to the heterogeneous database system in which the target database is located, so that the target transmission data is transmitted synchronously in several databases in the heterogeneous database system, the method further comprises the following steps:
acquiring preset local data format information;
and integrating the data formats of all databases in the heterogeneous database system into a data format matched with the data format information according to the local data format information.
6. The heterogeneous database synchronization method according to claim 5, wherein after the step of integrating the data formats of the databases in the heterogeneous database system into the data format matching the data format information according to the local data format information, the method further comprises the following steps:
acquiring change information of data in each database in the heterogeneous database system;
and generating a change data table of each database according to the change information.
7. The heterogeneous database synchronization method according to claim 1, wherein the step of transmitting the target transmission data to a heterogeneous database system so that the target transmission data is transmitted synchronously in several databases in the heterogeneous database system further comprises the steps of:
acquiring the database increment data in the target transmission data and a change record version corresponding to the database increment data;
comparing the version of the change record with the version of the local record;
and broadcasting the target transmission data when the change record version is matched with the local record version, so that a first target database of which the record version is not matched with the change record version in the heterogeneous database system carries out preset updating operation according to the target transmission data.
8. A heterogeneous database synchronization apparatus, comprising:
the first acquisition module is used for acquiring incremental data of the database;
the first processing module is used for packaging the database incremental data into target transmission data according to a preset custom packaging protocol;
the first execution module is used for transmitting the target transmission data to a heterogeneous database system so as to enable the target transmission data to be synchronously transmitted in a plurality of databases in the heterogeneous database system.
9. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the heterogeneous database synchronization method of any of claims 1 to 7.
10. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the heterogeneous database synchronization method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811644117.8A CN111382201B (en) | 2018-12-29 | 2018-12-29 | Heterogeneous database synchronization method, heterogeneous database synchronization device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811644117.8A CN111382201B (en) | 2018-12-29 | 2018-12-29 | Heterogeneous database synchronization method, heterogeneous database synchronization device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111382201A true CN111382201A (en) | 2020-07-07 |
| CN111382201B CN111382201B (en) | 2023-08-18 |
Family
ID=71216610
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811644117.8A Active CN111382201B (en) | 2018-12-29 | 2018-12-29 | Heterogeneous database synchronization method, heterogeneous database synchronization device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111382201B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112612763A (en) * | 2020-12-28 | 2021-04-06 | 北京浪潮数据技术有限公司 | Business file updating method, system and related components of file server |
| CN112883119A (en) * | 2021-04-30 | 2021-06-01 | 腾讯科技（深圳）有限公司 | Data synchronization method and device, computer equipment and computer readable storage medium |
| CN113434367A (en) * | 2021-06-30 | 2021-09-24 | 青岛海尔科技有限公司 | Data monitoring method and device, storage medium and electronic device |
| CN113434509A (en) * | 2021-07-02 | 2021-09-24 | 挂号网（杭州）科技有限公司 | Updating method and device of incremental index, storage medium and electronic equipment |
| CN114579591A (en) * | 2022-03-04 | 2022-06-03 | 常州市测绘院 | Geographic data multi-database collaborative updating maintenance method and maintenance system |
| CN114785864A (en) * | 2022-06-16 | 2022-07-22 | 联宝(合肥)电子科技有限公司 | Data interaction method, system, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106156094A (en) * | 2015-04-01 | 2016-11-23 | 阿里巴巴集团控股有限公司 | The teledata synchronous method of a kind of data base and device |
| CN107784016A (en) * | 2016-08-30 | 2018-03-09 | 北京京东尚科信息技术有限公司 | Data syn-chronization control device and data synchronous system and method |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
-
2018
- 2018-12-29 CN CN201811644117.8A patent/CN111382201B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106156094A (en) * | 2015-04-01 | 2016-11-23 | 阿里巴巴集团控股有限公司 | The teledata synchronous method of a kind of data base and device |
| CN107784016A (en) * | 2016-08-30 | 2018-03-09 | 北京京东尚科信息技术有限公司 | Data syn-chronization control device and data synchronous system and method |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112612763A (en) * | 2020-12-28 | 2021-04-06 | 北京浪潮数据技术有限公司 | Business file updating method, system and related components of file server |
| CN112883119A (en) * | 2021-04-30 | 2021-06-01 | 腾讯科技（深圳）有限公司 | Data synchronization method and device, computer equipment and computer readable storage medium |
| CN112883119B (en) * | 2021-04-30 | 2022-05-17 | 腾讯科技（深圳）有限公司 | Data synchronization method and device, computer equipment and computer readable storage medium |
| CN113434367A (en) * | 2021-06-30 | 2021-09-24 | 青岛海尔科技有限公司 | Data monitoring method and device, storage medium and electronic device |
| CN113434509A (en) * | 2021-07-02 | 2021-09-24 | 挂号网（杭州）科技有限公司 | Updating method and device of incremental index, storage medium and electronic equipment |
| CN114579591A (en) * | 2022-03-04 | 2022-06-03 | 常州市测绘院 | Geographic data multi-database collaborative updating maintenance method and maintenance system |
| CN114785864A (en) * | 2022-06-16 | 2022-07-22 | 联宝(合肥)电子科技有限公司 | Data interaction method, system, electronic equipment and storage medium |
| CN114785864B (en) * | 2022-06-16 | 2022-09-02 | 联宝(合肥)电子科技有限公司 | Data interaction method, system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111382201B (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111382201A (en) | Heterogeneous database synchronization method and device, computer equipment and storage medium | |
| CN108958714B (en) | Business component type development method and device, computer equipment and storage medium | |
| CN110162662B (en) | Verification method, device and equipment for data records in block chain type account book | |
| WO2021164462A1 (en) | Data encryption method, data decryption method, computer device, and medium | |
| CN109067732A (en) | Internet of things equipment and data insertion system, method and computer readable storage medium | |
| CN112486915A (en) | Data storage method and device | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN106406980A (en) | Deployment method and device for virtual machines | |
| CN109657167B (en) | Data acquisition method, device, server and storage medium | |
| CN105407114A (en) | Client and method for processing Zookeeper cluster data | |
| CN113704794A (en) | Bidding file processing method and device in electronic bidding system | |
| CN113515387B (en) | Data processing method and device and electronic device | |
| CN115756516A (en) | Model end-side deployment method, device, equipment and storage medium | |
| CN112667393B (en) | Method and device for building distributed task computing scheduling framework and computer equipment | |
| US11537455B2 (en) | Schema management using an event stream | |
| CN115129781A (en) | Data processing method and device, electronic equipment and storage medium | |
| US9952575B2 (en) | Energy management systems and methods | |
| CN114157662A (en) | Cloud platform parameter adaptation method and device, terminal equipment and storage medium | |
| CN106649720B (en) | Data processing method and server | |
| CN113572611B (en) | Key processing method and device and electronic device | |
| EP4383079A2 (en) | Schema and application code management using an event stream | |
| CN118092982B (en) | Multi-cluster operation and maintenance method, equipment and medium for cloud native application | |
| CN114547661B (en) | Encryption and decryption method, device, equipment and storage medium of application configuration data | |
| CN113918130A (en) | Data encapsulation method, device, application program server and storage medium | |
| CN110187870A (en) | A kind of shared integrated system based on business module |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |