CN110990482A - Data synchronization method and device between asynchronous databases - Google Patents
Data synchronization method and device between asynchronous databases Download PDFInfo
- Publication number
- CN110990482A CN110990482A CN201911092881.3A CN201911092881A CN110990482A CN 110990482 A CN110990482 A CN 110990482A CN 201911092881 A CN201911092881 A CN 201911092881A CN 110990482 A CN110990482 A CN 110990482A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- database
- synchronization
- data type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a method and a device for synchronizing data between asynchronous databases, wherein the method comprises the following steps: extracting a target statement for creating a target table from a source database, and converting the data type of the source database into the data type of a target database; applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table; and inserting the table data of the target table after format conversion into the target database in the form of a Python result set. According to the method and the device, data type conversion, data conversion and data loading can be automatically realized, the development workload in the required configuration and data synchronization process can be effectively reduced, and the cost of the data synchronization process can be effectively reduced.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a data synchronization method and device between asynchronous databases.
Background
Under the large background of removing IOE, more and more original systems using Oracle database or other commercial databases select object-relational database management system PostgreSQL or other open source or domestic databases using relational database management system MySQL, free software with very complete characteristics based on the consideration of cost or other factors. Whereas for software systems, switching databases necessarily involves migration of data.
At present, the data migration mode in heterogeneous databases is usually to select mature business software, such as data management software Informatica, data integration software platform DataStage or microsoft data transmission service DTS tool, etc., which is expensive and involves a certain amount of development; in order to solve this problem, data migration may be performed in a heterogeneous database by an automated means.
However, the existing method of performing data migration in heterogeneous databases by an automated means usually has a large amount of configuration contents, and the development workload in the data migration process is large, which brings a large migration cost to the data migration process.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a data synchronization method and device between asynchronous databases, which can automatically realize data type conversion, data conversion and data loading, effectively reduce the required configuration and development workload in the data synchronization process, and effectively reduce the cost of the data synchronization process.
In order to solve the technical problem, the application provides the following technical scheme:
in a first aspect, the present application provides a method for synchronizing data between asynchronous databases, including:
extracting a target statement for creating a target table from a source database, and converting the data type of the source database into the data type of a target database;
applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table;
and inserting the table data of the target table after format conversion into the target database in the form of a Python result set.
Further, before extracting the target statement for creating the target table from the source database, the method further includes:
acquiring a data type conversion configuration rule between the source database and the target database;
wherein the data type conversion configuration rule comprises: the unique identifier, connection information and data type of the source database, and the unique identifier, connection information, data type and data format of the target database.
Further, before extracting the target statement for creating the target table from the source database, the method further includes:
receiving a synchronization table list, wherein the synchronization table list is used for specifying target table information to be synchronized in a source database;
determining a corresponding at least one of the target tables based on target table information in the synchronization table manifest.
Further, the extracting a target statement for creating a target table from the source database includes:
connecting to the corresponding source database by applying the unique identifier and the connection information of the source database;
and extracting a target statement for creating a target table from the source database.
Further, the converting the data type of the source database into the data type of the target database includes:
determining a preset data conversion rule based on the data type of the source database and the data type of the target database;
and converting the data type of the source database into the data type of the target database according to the data conversion rule.
Further, the performing format conversion corresponding to the target database on the table data of the target table includes:
and converting the date format in the table data of the target table by applying the data format of the target database.
In a second aspect, the present application provides an apparatus for synchronizing data between asynchronous databases, comprising:
the data type conversion module is used for extracting a target statement for creating a target table from a source database and converting the data type of the source database into the data type of a target database;
the data format conversion module is used for establishing a table which accords with the data type of the target database in the target database by applying the target statement and performing format conversion corresponding to the target database on the table data of the target table;
and the data synchronization module is used for inserting the table data of the target table after format conversion into the target database in a Python result set form.
Further, still include:
the data type conversion configuration rule setting module is used for acquiring a data type conversion configuration rule between the source database and the target database;
wherein the data type conversion configuration rule comprises: the unique identifier, connection information and data type of the source database, and the unique identifier, connection information, data type and data format of the target database.
Further, still include:
the system comprises a synchronization table list receiving module, a synchronization table list receiving module and a synchronization table list sending module, wherein the synchronization table list is used for specifying target table information to be synchronized in a source database;
and the target table determining module is used for determining at least one corresponding target table based on the target table information in the synchronization table list.
Further, the data type conversion module comprises:
the source database connecting unit is used for applying the unique identifier and the connection information of the source database and connecting the source database to the corresponding source database;
and the target statement creating unit is used for extracting a target statement for creating a target table from the source database.
Further, the data type conversion module comprises:
a data conversion rule determining unit, configured to determine a preset data conversion rule based on the data type of the source database and the data type of the target database;
and the type conversion unit is used for converting the data type of the source database into the data type of the target database according to the data conversion rule.
Further, the data format conversion module includes:
and the date format conversion unit is used for converting the format of the date format in the table data of the target table by applying the data format of the target database.
In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data synchronization method between asynchronous databases when executing the program.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method for data synchronization between asynchronous databases.
According to the technical scheme, the data synchronization method and device between the asynchronous databases provided by the application extracts the target statement for creating the target table from the source database, and converts the data type of the source database into the data type of the target database; applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table; and inserting the table data of the target table after format conversion into the target database in a Python result set form, wherein the Python can be used as an intermediate medium, and data type conversion, data conversion and data loading are automatically realized through a preset heterogeneous database data type conversion rule. The data synchronization between the heterogeneous databases can be realized only by little configuration, the development workload in the data synchronization process is greatly reduced, and the method is a low-cost data synchronization solution. The synchronization scheme has the advantages that the programs are independently deployed during synchronization, the deployment at a source database end or a target database end is not needed, the coupling is less, and the synchronization scheme is suitable for synchronization among various mainstream heterogeneous databases.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data synchronization method between asynchronous databases in an embodiment of the present application.
Fig. 2 is a flowchart illustrating a data synchronization method between asynchronous databases including step 010 according to an embodiment of the present application.
Fig. 3 is a schematic flowchart of a data synchronization method between asynchronous databases including steps 020 and 030 in this embodiment of the present application.
Fig. 4 is a flowchart illustrating steps 111 and 112 in a data synchronization method between asynchronous databases in an embodiment of the present application.
Fig. 5 is a flowchart illustrating steps 121 and 122 in the data synchronization method between asynchronous databases in the embodiment of the present application.
Fig. 6 is a schematic flowchart of a data synchronization method between asynchronous databases in an application example of the present application.
Fig. 7 is a schematic diagram of a first structure of a data synchronization apparatus between asynchronous databases in an embodiment of the present application.
Fig. 8 is a schematic diagram of a second structure of a data synchronization apparatus between asynchronous databases in an embodiment of the present application.
Fig. 9 is a schematic diagram of a third structure of a data synchronization apparatus between asynchronous databases in an embodiment of the present application.
Fig. 10 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In consideration of the problems that a large amount of configuration content generally exists in the existing mode of carrying out data migration on heterogeneous databases through an automatic means, the development workload in the data process is large, and the migration cost is high in the data migration process, the application provides a data synchronization method among asynchronous databases, a data synchronization device among asynchronous databases, electronic equipment and a computer-readable storage medium, and by extracting target statements used for creating a target table from a source database, the data types of the source database are converted into the data types of a target database; applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table; and inserting the table data of the target table after format conversion into the target database in a Python result set form, wherein the Python can be used as an intermediate medium, and data type conversion, data conversion and data loading are automatically realized through a preset heterogeneous database data type conversion rule. The data synchronization between the heterogeneous databases can be realized only by little configuration, the development workload in the data synchronization process is greatly reduced, and the method is a low-cost data synchronization solution. The synchronization scheme has the advantages that the programs are independently deployed during synchronization, the deployment at a source database end or a target database end is not needed, the coupling is less, and the synchronization scheme is suitable for synchronization among various mainstream heterogeneous databases.
In one or more embodiments of the present application, Python is an object-oriented interpreted computer programming language.
In one or more embodiments of the present application, Oracle is also known under the full name Oracle Database, Oracle RDBMS, or Oracle for short. Is a relational database management system of the oracle culture company. It is a product that is always in the leading position in the field of databases.
In one or more embodiments of the present application, MySQL is one of the most popular Relational Database Management systems, and in terms of WEB applications, MySQL is one of the best RDBMS (Relational Database Management System) application software. Due to the characteristics of small size, high speed and low total cost of ownership, especially the development of source codes, MySQL is selected as a website database for the development of general small and medium-sized websites.
In one or more embodiments of the present application, PostgreSQL is a very full-featured, free-software object-relational database management system (ordms) that is well-compatible with Oracle databases. Because of the flexibility of the license, anyone can use, modify and distribute it for any purpose, free of charge.
In one or more embodiments of the present application, the intent of removing IOE is to remove IBM's minicomputer, Oracle database, EMC storage under the IT architecture of the original enterprise.
In one or more embodiments of the present application, information is a comprehensive, unified, open and economic data integration platform of the information company, and can support multiple complex enterprise-level data integration plans, including: enterprise Data integration, big Data, Data quality control, master Data management, B2B Data Exchange, application information lifecycle management, complex time processing, super messaging, and cloud Data integration.
In one or more embodiments of the present application, DataStage is a data integration software platform from IBM corporation that can help enterprises obtain more value from complex heterogeneous information spread across various systems.
In one or more embodiments of the present application, the DTS is a set of tools in microsoft SQL Server database that can transform heterogeneous data.
In view of the above, an embodiment of the present application provides a data synchronization apparatus between asynchronous databases, where the data synchronization apparatus between asynchronous databases may be a server, and the server may be communicatively connected to a plurality of asynchronous databases and may also be communicatively connected to at least one client device. The data synchronization device between the asynchronous databases can receive a data synchronization instruction between the asynchronous databases sent by the client device on line, then extract a target statement for creating a target table from a source database, and convert the data type of the source database into the data type of the target database; applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table; and finally, the data synchronization device between the asynchronous databases can also send the data synchronization result to the client equipment on line, so that a user can obtain the data synchronization result from the client equipment, and the intelligence degree of the data synchronization process between the asynchronous databases is further improved.
It is understood that the client devices may include smart phones, tablet electronic devices, network set-top boxes, portable computers, desktop computers, Personal Digital Assistants (PDAs), in-vehicle devices, smart wearable devices, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In practical applications, part of the data synchronization between the asynchronous databases may be performed on the server side as described above, or all operations may be performed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.
The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
The server and the client device may communicate using any suitable network protocol, including network protocols not yet developed at the filing date of this application. The network protocol may include, for example, a TCP/IP protocol, a UDP/IP protocol, an HTTP protocol, an HTTPS protocol, or the like. Of course, the network Protocol may also include, for example, an RPC Protocol (Remote Procedure Call Protocol), a REST Protocol (Representational State Transfer Protocol), and the like used above the above Protocol.
In order to automatically implement data type conversion, data conversion and data loading, effectively reduce the development workload in the required configuration and data synchronization process, and effectively reduce the cost of the data synchronization process, the present application provides an embodiment of a data synchronization method between asynchronous databases, and referring to fig. 1, the data synchronization method between asynchronous databases specifically includes the following contents:
step 100: and extracting a target statement for creating a target table from a source database, and converting the data type of the source database into the data type of the target database.
Step 200: and applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table.
Step 300: and inserting the table data of the target table after format conversion into the target database in the form of a Python result set.
In order to automatically implement data type conversion, data conversion and data loading, and further improve the synchronization efficiency of the data synchronization process between asynchronous databases, in an embodiment of the data synchronization method between asynchronous databases of the present application, referring to fig. 2, step 100 in the data synchronization method between asynchronous databases further includes the following steps:
step 010: acquiring a data type conversion configuration rule between the source database and the target database;
wherein the data type conversion configuration rule comprises: the unique identifier, connection information and data type of the source database, and the unique identifier, connection information, data type and data format of the target database.
In order to automatically implement data type conversion, data conversion and data loading, and further improve the pertinence and intelligence of the data synchronization process between asynchronous databases, in an embodiment of the data synchronization method between asynchronous databases of the present application, referring to fig. 3, step 100 in the data synchronization method between asynchronous databases further includes the following steps:
step 020: receiving a synchronization table list, wherein the synchronization table list is used for specifying target table information to be synchronized in a source database.
Step 030: determining a corresponding at least one of the target tables based on target table information in the synchronization table manifest.
Therefore, the data synchronization method between asynchronous databases provided by the embodiment of the application can adapt to synchronization of all tables of the source database, and can also realize synchronization of partial tables of the source database.
In order to improve the extraction efficiency and accuracy of the target statement for creating the target table, in an embodiment of the data synchronization method between asynchronous databases of the present application, referring to fig. 4, step 100 in the data synchronization method between asynchronous databases specifically includes the following contents:
step 111: and connecting to the corresponding source database by applying the unique identifier and the connection information of the source database.
Step 112: and extracting a target statement for creating a target table from the source database.
In order to improve the efficiency and accuracy of data type conversion, in an embodiment of the data synchronization method between asynchronous databases of the present application, referring to fig. 5, step 100 in the data synchronization method between asynchronous databases further includes the following steps:
step 121: and determining a preset data conversion rule based on the data type of the source database and the data type of the target database.
Step 122: and converting the data type of the source database into the data type of the target database according to the data conversion rule.
In order to improve the efficiency and accuracy of format conversion, in an embodiment of the data synchronization method between asynchronous databases of the present application, step 200 in the data synchronization method between asynchronous databases further includes the following steps:
step 210: and converting the date format in the table data of the target table by applying the data format of the target database.
In order to further implement the present solution, the present application further provides a specific application example of a data synchronization method between asynchronous databases, and referring to fig. 6, the data synchronization method between asynchronous databases specifically includes the following contents:
data type conversion setting
The data type conversion relationship can be configured through the interactive interface, so that the method has the advantages of simplicity and less configuration.
The method mainly comprises a source database type, a target database type, a source data type, a target data type, source database connection information, target database connection information and the like, and the source database type, the target database type, the source database connection information, the target database connection information and the like are stored in corresponding configuration tables for synchronous use.
The specific configuration content is as follows:
(1) source database name: storing a source database name, such as Oracle;
(2) target database name: storing the name of the target database, such as MySQL;
(3) source database connection information: connection information of a source database;
(4) target database connection information: connection information of the target database;
(5) the source data type: such as character type, numerical type, etc.;
(6) target data type: such as character type, numerical type, etc.;
(7) target data format: 4 years 2 months 2 days (YYYYMMDD).
The preset data type mapping rule of Oracle and MySQL is shown in Table 1:
TABLE 1
(II) synchronization Table rule set
And configuring a synchronous table list through an interactive interface, namely setting the table list and table data filter conditions which need to be synchronized in the source database, and if not, defaulting to be all. Namely:
(1) configuring a table list needing synchronization and filtering conditions;
(2) or not configured, not configured as full-library synchronization;
after the synchronization parameters are configured, the data are synchronized from the source database to the target database through the synchronization execution function. And the type conversion and the format conversion are automatically completed in the synchronization process.
(III) synchronous execution
By executing the synchronization function, the synchronization is realized rapidly in batches.
1) And performing data synchronization of the designated table or all tables according to the selected table list.
The synchronization of the tables includes the conversion of data types and data formats, and the whole synchronization process is illustrated by a single table.
2) And connecting to the source database according to the connection information of the configuration database, and extracting the statement for creating the table.
3) And converting the data type of the source database into the data type of the target database according to the set data conversion rule.
4) And creating a table which accords with the data type of the target database in the target database, and completing the conversion of the table structure.
5) Conversion and synchronization of data:
after the data type conversion is completed, the data conversion is mainly the conversion of date format, for example, yyy-mm-dd is the format of Oracle database, and is another format in MySQL, and by this step, the data format is converted from the source database to the format meeting the requirement of the target database.
And inserting the converted data into a target database in a Python result set mode, thereby completing the conversion and synchronization of the data. I.e. all based on Python implementation, making it lightweight.
As can be seen from the above description, the present application is based on the Python technology, uses Python as an intermediate medium, and automatically implements data type conversion, data conversion, and data loading through a preset heterogeneous database data type conversion rule. The data synchronization between the heterogeneous databases can be realized only by little configuration, the development workload in the data synchronization process is greatly reduced, and the method is a low-cost data synchronization solution. The synchronization scheme has the advantages that the programs are independently deployed during synchronization, the deployment at a source database end or a target database end is not needed, the coupling is less, and the synchronization scheme is suitable for synchronization among various mainstream heterogeneous databases. The method has the characteristics of low cost, small development amount and easy realization.
In order to automatically implement data type conversion, data conversion and data loading, effectively reduce the development workload in the required configuration and data synchronization process, and effectively reduce the cost of the data synchronization process, the present application provides an embodiment of a data synchronization apparatus between asynchronous databases for implementing all or part of the contents in the data synchronization method between the asynchronous databases, and referring to fig. 7, the data synchronization apparatus between the asynchronous databases specifically includes the following contents:
the data type conversion module 10 is configured to extract a target statement for creating a target table from a source database, and convert a data type of the source database into a data type of a target database.
And the data format conversion module 20 is configured to apply the target statement to create a table in the target database, where the table conforms to the data type of the target database, and perform format conversion on table data of the target table corresponding to the target database.
And a data synchronization module 30, configured to insert the table data of the target table after format conversion into the target database in a Python result set.
In order to automatically implement data type conversion, data conversion and data loading, and further improve the synchronization efficiency of the data synchronization process between asynchronous databases, in an embodiment of the data synchronization device between asynchronous databases of the present application, referring to fig. 8, the data synchronization device between asynchronous databases further includes the following contents:
a data type conversion configuration rule setting module 01, configured to obtain a data type conversion configuration rule between the source database and the target database;
wherein the data type conversion configuration rule comprises: the unique identifier, connection information and data type of the source database, and the unique identifier, connection information, data type and data format of the target database.
In order to automatically implement data type conversion, data conversion and data loading, and further improve the pertinence and intelligence of the data synchronization process between asynchronous databases, in an embodiment of the data synchronization device between asynchronous databases of the present application, referring to fig. 9, the data synchronization device between asynchronous databases further includes the following contents:
the synchronization table list receiving module 02 is configured to receive a synchronization table list, where the synchronization table list is used to specify target table information to be synchronized in a source database.
And the target table determining module 03 is configured to determine, based on target table information in the synchronization table list, at least one corresponding target table.
In order to improve the extraction efficiency and accuracy of the target statements used for creating the target table, in an embodiment of the data synchronization apparatus between asynchronous databases of the present application, the data type conversion module 10 in the data synchronization apparatus between asynchronous databases specifically includes the following contents:
the source database connecting unit is used for applying the unique identifier and the connection information of the source database and connecting the source database to the corresponding source database;
and the target statement creating unit is used for extracting a target statement for creating a target table from the source database.
In order to improve the efficiency and accuracy of data type conversion, in an embodiment of the data synchronization apparatus between asynchronous databases of the present application, the data type conversion module 10 in the data synchronization apparatus between asynchronous databases further includes the following contents:
a data conversion rule determining unit, configured to determine a preset data conversion rule based on the data type of the source database and the data type of the target database;
and the type conversion unit is used for converting the data type of the source database into the data type of the target database according to the data conversion rule.
In order to improve the efficiency and accuracy of format conversion, in an embodiment of the data synchronization apparatus between asynchronous databases of the present application, the data format conversion module 20 in the data synchronization apparatus between asynchronous databases further includes the following contents:
and the date format conversion unit is used for converting the format of the date format in the table data of the target table by applying the data format of the target database.
In terms of hardware, in order to automatically implement data type conversion, data conversion, and data loading, effectively reduce the development workload in the required configuration and data synchronization process, and effectively reduce the cost of the data synchronization process, the present application provides an embodiment of an electronic device for implementing all or part of the contents in the data synchronization method between asynchronous databases, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between a data synchronization device among asynchronous databases and a plurality of asynchronous databases, user terminals and other related equipment; the electronic device may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the electronic device may refer to an embodiment of a data synchronization method between asynchronous databases and an embodiment of a data synchronization apparatus between asynchronous databases in the embodiment, and the contents of the embodiments are incorporated herein, and repeated details are not repeated.
Fig. 10 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 10, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 10 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the data synchronization function between asynchronous databases may be integrated into the central processor 9100. The central processor 9100 may be configured to control as follows:
step 100: and extracting a target statement for creating a target table from a source database, and converting the data type of the source database into the data type of the target database.
Step 200: and applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table.
Step 300: and inserting the table data of the target table after format conversion into the target database in the form of a Python result set.
As can be seen from the above description, the electronic device provided in the embodiment of the present application can use Python as an intermediary, and automatically implement data type conversion, data conversion, and data loading through a preset heterogeneous database data type conversion rule. The data synchronization between the heterogeneous databases can be realized only by little configuration, the development workload in the data synchronization process is greatly reduced, and the method is a low-cost data synchronization solution. The synchronization scheme has the advantages that the programs are independently deployed during synchronization, the deployment at a source database end or a target database end is not needed, the coupling is less, and the synchronization scheme is suitable for synchronization among various mainstream heterogeneous databases.
In another embodiment, the data synchronization device between the asynchronous databases may be configured separately from the central processor 9100, for example, the data synchronization device between the asynchronous databases may be configured as a chip connected to the central processor 9100, and the data synchronization function between the asynchronous databases may be implemented by the control of the central processor.
As shown in fig. 10, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 10; in addition, the electronic device 9600 may further include components not shown in fig. 10, which can be referred to in the prior art.
As shown in fig. 10, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
Embodiments of the present application further provide a computer-readable storage medium capable of implementing all steps in the data synchronization method between asynchronous databases in the foregoing embodiments, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all steps of the data synchronization method between asynchronous databases, where the execution subject is a server or a client, for example, when the processor executes the computer program, the processor implements the following steps:
step 100: and extracting a target statement for creating a target table from a source database, and converting the data type of the source database into the data type of the target database.
Step 200: and applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table.
Step 300: and inserting the table data of the target table after format conversion into the target database in the form of a Python result set.
As can be seen from the above description, the computer-readable storage medium provided in the embodiment of the present application can use Python as an intermediate medium, and automatically implement data type conversion, data conversion, and data loading through a preset heterogeneous database data type conversion rule. The data synchronization between the heterogeneous databases can be realized only by little configuration, the development workload in the data synchronization process is greatly reduced, and the method is a low-cost data synchronization solution. The synchronization scheme has the advantages that the programs are independently deployed during synchronization, the deployment at a source database end or a target database end is not needed, the coupling is less, and the synchronization scheme is suitable for synchronization among various mainstream heterogeneous databases.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (14)
1. A method for synchronizing data between asynchronous databases, comprising:
extracting a target statement for creating a target table from a source database, and converting the data type of the source database into the data type of a target database;
applying the target statement to create a table which accords with the data type of the target database in the target database, and performing format conversion corresponding to the target database on table data of the target table;
and inserting the table data of the target table after format conversion into the target database in the form of a Python result set.
2. The method for data synchronization between asynchronous databases as claimed in claim 1, further comprising, before extracting the target statement for creating the target table from the source database:
acquiring a data type conversion configuration rule between the source database and the target database;
wherein the data type conversion configuration rule comprises: the unique identifier, connection information and data type of the source database, and the unique identifier, connection information, data type and data format of the target database.
3. The method for data synchronization between asynchronous databases as claimed in claim 1, further comprising, before extracting the target statement for creating the target table from the source database:
receiving a synchronization table list, wherein the synchronization table list is used for specifying target table information to be synchronized in a source database;
determining a corresponding at least one of the target tables based on target table information in the synchronization table manifest.
4. The method for data synchronization between asynchronous databases as claimed in claim 2, wherein the extracting of the target statement for creating the target table from the source database comprises:
connecting to the corresponding source database by applying the unique identifier and the connection information of the source database;
and extracting a target statement for creating a target table from the source database.
5. The method for data synchronization between asynchronous databases as claimed in claim 2, wherein said converting the data type of the source database into the data type of the target database comprises:
determining a preset data conversion rule based on the data type of the source database and the data type of the target database;
and converting the data type of the source database into the data type of the target database according to the data conversion rule.
6. The method for data synchronization between asynchronous databases according to claim 2, wherein said performing format conversion on table data of the target table corresponding to the target database comprises:
and converting the date format in the table data of the target table by applying the data format of the target database.
7. An apparatus for synchronizing data between asynchronous databases, comprising:
the data type conversion module is used for extracting a target statement for creating a target table from a source database and converting the data type of the source database into the data type of a target database;
the data format conversion module is used for establishing a table which accords with the data type of the target database in the target database by applying the target statement and performing format conversion corresponding to the target database on the table data of the target table;
and the data synchronization module is used for inserting the table data of the target table after format conversion into the target database in a Python result set form.
8. The apparatus for data synchronization between asynchronous databases according to claim 7, further comprising:
the data type conversion configuration rule setting module is used for acquiring a data type conversion configuration rule between the source database and the target database;
wherein the data type conversion configuration rule comprises: the unique identifier, connection information and data type of the source database, and the unique identifier, connection information, data type and data format of the target database.
9. The apparatus for data synchronization between asynchronous databases according to claim 7, further comprising:
the system comprises a synchronization table list receiving module, a synchronization table list receiving module and a synchronization table list sending module, wherein the synchronization table list is used for specifying target table information to be synchronized in a source database;
and the target table determining module is used for determining at least one corresponding target table based on the target table information in the synchronization table list.
10. The apparatus for data synchronization between asynchronous databases of claim 8, wherein the data type conversion module comprises:
the source database connecting unit is used for applying the unique identifier and the connection information of the source database and connecting the source database to the corresponding source database;
and the target statement creating unit is used for extracting a target statement for creating a target table from the source database.
11. The apparatus for data synchronization between asynchronous databases of claim 8, wherein the data type conversion module comprises:
a data conversion rule determining unit, configured to determine a preset data conversion rule based on the data type of the source database and the data type of the target database;
and the type conversion unit is used for converting the data type of the source database into the data type of the target database according to the data conversion rule.
12. The apparatus for data synchronization between asynchronous databases of claim 8, wherein the data format conversion module comprises:
and the date format conversion unit is used for converting the format of the date format in the table data of the target table by applying the data format of the target database.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the program, implements the steps of the method for data synchronization between asynchronous databases of any of claims 1 to 6.
14. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method for data synchronization between asynchronous databases of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911092881.3A CN110990482A (en) | 2019-11-11 | 2019-11-11 | Data synchronization method and device between asynchronous databases |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911092881.3A CN110990482A (en) | 2019-11-11 | 2019-11-11 | Data synchronization method and device between asynchronous databases |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110990482A true CN110990482A (en) | 2020-04-10 |
Family
ID=70083655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911092881.3A Pending CN110990482A (en) | 2019-11-11 | 2019-11-11 | Data synchronization method and device between asynchronous databases |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990482A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666338A (en) * | 2020-05-26 | 2020-09-15 | 中国工商银行股份有限公司 | Data copying method, control node and electronic equipment |
| CN111930842A (en) * | 2020-08-10 | 2020-11-13 | 中国工商银行股份有限公司 | Data checking method and device |
| CN112256794A (en) * | 2020-10-09 | 2021-01-22 | 国网江西省电力有限公司检修分公司 | Database synchronization method |
| CN112380186A (en) * | 2020-11-06 | 2021-02-19 | 福建亿榕信息技术有限公司 | Domestic database migration method based on active adaptation |
| CN112632190A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | Data synchronization method and device |
| CN112783960A (en) * | 2021-01-22 | 2021-05-11 | 北京澎思科技有限公司 | Data processing system and method, electronic device, computer-readable storage medium |
| CN113760962A (en) * | 2021-09-01 | 2021-12-07 | 中国人民银行清算总中心 | Single-domain to cross-domain data set data processing method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN106649378A (en) * | 2015-11-02 | 2017-05-10 | 北大方正集团有限公司 | Data synchronizing method and device |
| CN108845921A (en) * | 2018-05-30 | 2018-11-20 | 郑州云海信息技术有限公司 | A kind of method, apparatus, readable storage medium storing program for executing and the computer equipment of test database performance |
| CN109656934A (en) * | 2018-11-19 | 2019-04-19 | 武汉达梦数据库有限公司 | Source oracle database DDL synchronous method and equipment based on log parsing |
| CN109783555A (en) * | 2018-12-13 | 2019-05-21 | 东软集团股份有限公司 | Form templat storage method, device, storage medium and electronic equipment |
| CN109992595A (en) * | 2019-04-11 | 2019-07-09 | 北京启迪区块链科技发展有限公司 | Different database conversion method, apparatus, equipment and storage medium |
-
2019
- 2019-11-11 CN CN201911092881.3A patent/CN110990482A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN106649378A (en) * | 2015-11-02 | 2017-05-10 | 北大方正集团有限公司 | Data synchronizing method and device |
| CN108845921A (en) * | 2018-05-30 | 2018-11-20 | 郑州云海信息技术有限公司 | A kind of method, apparatus, readable storage medium storing program for executing and the computer equipment of test database performance |
| CN109656934A (en) * | 2018-11-19 | 2019-04-19 | 武汉达梦数据库有限公司 | Source oracle database DDL synchronous method and equipment based on log parsing |
| CN109783555A (en) * | 2018-12-13 | 2019-05-21 | 东软集团股份有限公司 | Form templat storage method, device, storage medium and electronic equipment |
| CN109992595A (en) * | 2019-04-11 | 2019-07-09 | 北京启迪区块链科技发展有限公司 | Different database conversion method, apparatus, equipment and storage medium |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666338A (en) * | 2020-05-26 | 2020-09-15 | 中国工商银行股份有限公司 | Data copying method, control node and electronic equipment |
| CN111666338B (en) * | 2020-05-26 | 2023-08-15 | 中国工商银行股份有限公司 | Data replication method, control node and electronic equipment |
| CN111930842A (en) * | 2020-08-10 | 2020-11-13 | 中国工商银行股份有限公司 | Data checking method and device |
| CN111930842B (en) * | 2020-08-10 | 2024-02-09 | 中国工商银行股份有限公司 | Data checking method and device |
| CN112256794A (en) * | 2020-10-09 | 2021-01-22 | 国网江西省电力有限公司检修分公司 | Database synchronization method |
| CN112380186A (en) * | 2020-11-06 | 2021-02-19 | 福建亿榕信息技术有限公司 | Domestic database migration method based on active adaptation |
| CN112632190A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | Data synchronization method and device |
| CN112783960A (en) * | 2021-01-22 | 2021-05-11 | 北京澎思科技有限公司 | Data processing system and method, electronic device, computer-readable storage medium |
| CN113760962A (en) * | 2021-09-01 | 2021-12-07 | 中国人民银行清算总中心 | Single-domain to cross-domain data set data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110990482A (en) | Data synchronization method and device between asynchronous databases | |
| WO2018219178A1 (en) | Data synchronization method and apparatus, server, and storage medium | |
| CN111031058A (en) | Websocket-based distributed server cluster interaction method and device | |
| CN110413673B (en) | Database data unified acquisition and distribution method and system | |
| WO2019134306A1 (en) | Message processing method, device, terminal apparatus, and readable storage medium | |
| CN110457256A (en) | Date storage method, device, computer equipment and storage medium | |
| CN110990228A (en) | Data interface monitoring method and device | |
| CN111240858B (en) | Event scheduling method and component | |
| CN103139157A (en) | Network communication method based on socket, device and system | |
| CN103369034A (en) | Method, system and device for sending photos to digital photo frames | |
| CN105530630B (en) | A kind of OTA upgrade method and equipment | |
| CN105656643B (en) | Method and equipment for performing compatible management based on network configuration protocol | |
| CN113485952A (en) | Data batch transmission method and device | |
| CN112597199B (en) | Heterogeneous multi-data source adaptation method and device | |
| CN104298757A (en) | Method and system allowing compatibility with mobile clients and databases different in version | |
| CN110889765B (en) | Transaction information reporting method and device | |
| CN113392091A (en) | Distributed cluster data migration method and device | |
| CN112734545A (en) | Block chain data sharing method, device and system | |
| CN104320695A (en) | Method and system for family socializing of mobile device based on smart TV | |
| CN115082038A (en) | System integration method and device and electronic equipment | |
| CN113918436A (en) | Log processing method and device | |
| CN109426571B (en) | Method, system, storage medium, processor and apparatus for function call and data access | |
| CN113342501B (en) | System fault processing method and device | |
| CN115580622B (en) | Workflow execution method based on edge synchronization service, electronic equipment and medium | |
| CN112769589B (en) | Base station upgrading method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220915 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |