CN113032480A - Data synchronization method, system and device - Google Patents
Data synchronization method, system and device Download PDFInfo
- Publication number
- CN113032480A CN113032480A CN201911356636.9A CN201911356636A CN113032480A CN 113032480 A CN113032480 A CN 113032480A CN 201911356636 A CN201911356636 A CN 201911356636A CN 113032480 A CN113032480 A CN 113032480A
- Authority
- CN
- China
- Prior art keywords
- target
- database
- data
- information
- source database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000005540 biological transmission Effects 0.000 claims abstract description 18
- 230000003213 activating effect Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data synchronization method, system and device, which relate to the technical field of electronic information and can solve the problem of low efficiency when different kinds of databases transmit data. The specific technical scheme is as follows: acquiring a target configuration file, wherein the target configuration file can comprise target data, configuration information of a target database and configuration information of a source database; and generating target request information according to the target configuration file, requesting the source database to send target data, and after acquiring the target data, converting the target data and then sending the converted target data to the target database by the server, thereby realizing the technical effects of data transmission and data synchronization between the target database and the source database. The present disclosure is used for data synchronization between heterogeneous databases.
Description
Technical Field
The present disclosure relates to the field of electronic information technologies, and in particular, to a data synchronization method, system, and apparatus.
Background
With the rapid development of the big data era, the transmission technology between databases is also rapidly developed. In the prior art, when data needs to be transmitted among different data, the data to be transmitted is transmitted among databases by generating corresponding configuration information. When data synchronization is required to be realized among a plurality of data, the problem of increased manual workload and poor data synchronization accuracy is caused because a large amount of configuration information needs to be configured.
Disclosure of Invention
The embodiment of the disclosure provides a data synchronization method, a system and a device, which can solve the problem of low data synchronization efficiency between databases. The technical scheme is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a data synchronization method, including:
applied to a server, the method comprises the following steps:
obtaining a target configuration file, wherein the target configuration file comprises: configuration information of the source database, configuration information of the target database and identification information of the target data;
generating target request information according to the target configuration file and sending the target request information to the source database, wherein the request information is used for requesting the source database to send target data;
and obtaining a feedback result according to the target request information, and sending the feedback result to the target database.
In one embodiment, before generating the target request information, the method further includes:
judging whether the type of the source database and the type of the target database simultaneously accord with a preset database type according to the configuration information of the source database and the configuration information of the target database;
and when the type of the source database and the type of the target database simultaneously accord with the preset database type, starting a corresponding database processor, and activating a generating function, wherein the generating function is used for generating target request information.
In one embodiment, the method for feeding back the result includes a first feedback result and the obtaining the feedback result includes:
when the feedback result shows that the target data is successfully acquired, acquiring the target data;
and converting the target data according to a preset data format, and sending the converted target data to the target database.
In one embodiment, the method for obtaining the feedback result includes:
and when the feedback result does not include that the target data is not successfully acquired, generating target information and sending the target information to the target database, wherein the target information is used for indicating that the target data is not acquired from the source database.
In one embodiment, the method for generating the target request information comprises the following steps:
analyzing the target configuration file, and acquiring target address information and processing rules according to the analysis result;
respectively establishing a transmission link with the source database and a transmission link with the target database according to the target address information;
and determining the target data through the processing rule and the identification information of the target data.
The data synchronization method provided by the embodiment of the disclosure is applied to a server, and a target configuration file is obtained, wherein the target configuration file can comprise target data, configuration information of a target database and configuration information of a source database; and generating target request information according to the target configuration file, requesting a source database to send target data, and after acquiring the target data, converting the target data and then sending the converted target data to the target database by the server, thereby realizing data transmission and data synchronization between the target database and the source database.
According to a second aspect of the embodiments of the present disclosure, there is provided a data synchronization system, including:
a source database, a target database and a server,
the server is respectively connected with the source database and the target database;
the server is configured to obtain a target configuration file, where the target configuration file includes: configuration information of the source database, configuration information of the target database and identification information of the target data;
generating target request information according to the target configuration file, wherein the request information is used for requesting the source database to send target data;
sending the target request information to the source database; and obtaining a feedback result according to the target request information, and sending the feedback result to the target database.
The source database is used for acquiring target data according to the target request data and sending the target data to the server;
the target database is used for receiving the feedback result.
In one embodiment, the source database in the system is further configured to:
determining whether the source database can find the target data according to the target request information;
when the target data is found, generating a feedback result according to the target data;
when the target data is not found, generating target information, and generating a feedback result according to the target information, wherein the target information is used for indicating that the target data is not obtained from the source database;
and sends the feedback result to the server.
According to a third aspect of embodiments of the present disclosure, there is provided a server, including: a transmitter, a processor and a receiver, wherein,
the processor is respectively connected with the transmitter and the receiver;
the receiver is configured to obtain a target profile, where the target profile includes: configuration information of the source database, configuration information of the target database and identification information of the target data;
the processor is used for generating target request information according to the target configuration file and sending the target request information to the source database, wherein the request information is used for requesting the source database to send target data;
the transmitter is used for acquiring a feedback result according to the target request information and transmitting the feedback result to the target database.
In one embodiment, the processor in the server is further configured to,
when the feedback result shows that the target data is successfully acquired, acquiring the target data;
and converting the target data according to a preset data format, and sending the converted target data to the target database.
In one embodiment, the processor in the server is further configured to,
analyzing the target configuration file, and acquiring target address information and processing rules according to the analysis result;
respectively establishing a transmission link with the source database and a transmission link with the target database according to the target address information;
and determining the target data through the processing rule and the identification information of the target data.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a flowchart of a data synchronization method provided by an embodiment of the present disclosure;
FIG. 2 is a block diagram of a data synchronization system provided by an embodiment of the present disclosure;
fig. 3 is a block diagram of a server 30 provided in the embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
Example 1
An embodiment of the present disclosure provides a data synchronization method, as shown in fig. 1, the data synchronization method includes the following steps:
101. and acquiring a target configuration file.
The target configuration file includes: configuring information of a source database, configuring information of a target database and identification information of target data, wherein the configuring information comprises address information of the database, type of the database and table structure information in the database
In an alternative embodiment, the identification information of the target data in the target configuration file obtained by the method of the present disclosure may be obtained by including header information in a data table.
In an alternative embodiment, the type of the source database may be a database of SQL Server type, oracle type, MySQL type, or the like; the type of the target database can be a database of SQL Server type, Orcale type, MySQL type, etc.
102. And generating target request information according to the target configuration file, and sending the target request information to the source database.
The target request information is used for requesting the source database to send target data, the server sends the target data to the target database after acquiring the target data, and therefore data synchronization between the target database and the source database is achieved.
The method for generating the target request information comprises the following steps:
analyzing the target configuration file, and acquiring target address information and processing rules according to the analysis result;
respectively establishing a transmission link with the source database and a transmission link with the target database according to the target address information;
and determining the target data through the processing rule and the identification information of the target data.
If the processing rule is that when the data of the source database is detected to be updated, the updated data is determined to be the target data. And determining which specific type of data in the updated data is the target data according to the identification information.
Further, before generating the target request information, the method provided by the present disclosure further includes: and judging whether the database types of the source and destination databases configured in the configuration file are the database types supported by the system or not according to the read data.
And when the types of the source database and the target database meet the requirements, starting a corresponding database processor according to the database type of the task configuration file, generating corresponding data query request information according to the configuration information of the configuration file, and sending the data query request information to the source database so as to command the source database to execute a data query instruction.
And when the type of the source database and the type of the target database do not meet the requirements, generating prompt information, respectively sending the information to the source database and the target database, wherein the prompt information is used for prompting that the data types are not supported, and quitting the task execution.
103. And obtaining a feedback result according to the target request information, and sending the feedback result to the target database.
The obtaining of the feedback result in the method of the present disclosure includes:
when the feedback result shows that the target data is successfully acquired, acquiring the target data;
and converting the target data according to a preset data format, and sending the converted target data to the target database.
The preset data format can be set according to the format of the target database, and also can be set according to the format compatible with the target database and the source database, so that the target database can successfully receive the target data and store the target data.
On the other hand, the method for obtaining the feedback result includes:
and when the feedback result does not include that the target data is not successfully acquired, generating target information and sending the target information to the target database, wherein the target information is used for indicating that the target data is not acquired from the source database.
The data synchronization method provided by the embodiment of the disclosure is applied to a server, and a target configuration file is obtained, wherein the target configuration file can comprise target data, configuration information of a target database and configuration information of a source database; and generating target request information according to the target configuration file, requesting the source database to send target data, and after acquiring the target data, converting the target data and then sending the converted target data to the target database by the server, thereby realizing the technical effects of data transmission and data synchronization between the target database and the source database.
The data synchronization method can generate corresponding data transmission request information according to the task configuration file, the tasks needing data synchronization are configured in the configuration file according to different relational data sources, and the system carries out task scheduling according to the configuration information of the tasks, so that the aim of data synchronization among different databases is fulfilled.
Furthermore, the method provided by the disclosure can also be applied to customized development, in the development and operation and maintenance stage in the prior art, when a multi-use scene needs to be accessed, a large amount of manpower and material resources need to be invested, more cost needs to be invested in the operation and maintenance, the efficiency is low, when a large number of data tables are involved, omission and errors are prone to occur, if the data tables need to be synchronized regularly, an operator forgets easily, and the data tables or fields are difficult to deal with frequent changes.
Example two
Based on the data synchronization method described in the embodiment corresponding to fig. 1, the following is an embodiment of the system of the present disclosure, which may be used to execute an embodiment of the method of the present disclosure.
An embodiment of the present disclosure provides a data synchronization system, and as shown in fig. 2, the data synchronization system 20 includes: a source database 201, a target database 202 and a server 203,
the server 203 is connected to the source database 201 and the target database 202 respectively;
the server 203 is configured to obtain a target configuration file, where the target configuration file includes: configuration information of the source database, configuration information of the target database and identification information of the target data;
generating target request information according to the target configuration file, wherein the request information is used for requesting the source database to send target data;
sending the target request information to the source database 201;
and accepts the feedback result sent by the source database 201.
The source database 201 is configured to obtain target data according to the target request data, and send the target data to the server;
the target database 202 is used for receiving the feedback result.
In one embodiment, the source database in the system provided by the present disclosure is further configured to:
determining whether the source database can find the target data according to the target request information;
when the target data is found, generating a feedback result according to the target data;
when the target data is not found, generating target information, and generating a feedback result according to the target information, wherein the target information is used for indicating that the target data is not obtained from the source database;
and sends the feedback result to the server.
In one embodiment, the target database in the system provided by the present disclosure is further configured to:
receiving the feedback result;
analyzing the feedback result, and storing the target data into a target data table when the feedback result comprises the target data;
when the feedback result comprises target information, generating a feedback result according to the target information, wherein the target information is used for indicating that target data is not acquired from the source database;
and sends the feedback result to the server.
In the data synchronization system provided by the embodiment of the present disclosure, a server in the system acquires a target configuration file, where the target configuration file may include target data, configuration information of a target database, and configuration information of a source database; and after obtaining the feedback result, the server converts the feedback result and sends the converted feedback result to the target database, thereby realizing the technical effects of data transmission and data synchronization between the target database and the source database.
EXAMPLE III
Based on the data synchronization method described in the embodiment corresponding to fig. 1, the following is an embodiment of the apparatus of the present disclosure, which may be used to execute an embodiment of the method of the present disclosure.
The present disclosure provides a server 30 comprising: a receiver 301, a processor 302 and a transmitter 303,
the processor 302 is connected to the transmitter 303 and the receiver 301, respectively;
the receiver 301 is configured to obtain a target profile, where the target profile includes: configuration information of the source database, configuration information of the target database and identification information of the target data;
the processor 302 is configured to generate target request information according to the target configuration file and send the target request information to the source database, where the target request information is used to request the source database to send target data;
the transmitter 303 is configured to obtain a feedback result according to the target request information, and transmit the feedback result to the target database.
The present disclosure provides a server, wherein the processor 302 is further configured to,
when the feedback result shows that the target data is successfully acquired, acquiring the target data;
and converting the target data according to a preset data format, and sending the converted target data to the target database.
The present disclosure provides a server, wherein the processor 302 is further configured to,
analyzing the target configuration file, and acquiring target address information and processing rules according to the analysis result;
respectively establishing a transmission link with the source database and a transmission link with the target database according to the target address information;
and determining the target data through the processing rule and the identification information of the target data.
Based on the data synchronization method described in the embodiment corresponding to fig. 1 and fig. 2, an embodiment of the present disclosure further provides a computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a Read Only Memory (ROM), a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. The storage medium stores computer instructions for executing the data synchronization method described in the embodiment corresponding to fig. 1 and fig. 2, which is not described herein again.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (10)
1. A method for data synchronization, applied to a server, the method comprising:
obtaining a target configuration file, wherein the target configuration file comprises: configuration information of the source database, configuration information of the target database and identification information of the target data;
generating target request information and sending the target request information to the source database according to the target configuration file, wherein the target request information is used for requesting the source database to send target data;
and obtaining a feedback result according to the target request information, and sending the feedback result to the target database.
2. The method of claim 1, wherein prior to generating the target request message, the method further comprises:
judging whether the type of the source database and the type of the target database simultaneously accord with a preset database type according to the configuration information of the source database and the configuration information of the target database;
and when the type of the source database and the type of the target database simultaneously accord with a preset database type, starting a corresponding database processor, and activating a generating function, wherein the generating function is used for generating target request information.
3. The method of claim 1, wherein the feedback results comprise first feedback results and the obtaining feedback results comprise:
when the feedback result shows that the target data is successfully acquired, acquiring the target data;
and converting the target data according to a preset data format, and sending the converted target data to the target database.
4. The method of claim 3, wherein obtaining the feedback result comprises:
and when the feedback result does not include that the target data is not successfully acquired, generating target information and sending the target information to the target database, wherein the target information is used for indicating that the target data is not acquired from the source database.
5. The method of claim 1, wherein generating the target request message comprises:
analyzing the target configuration file, and acquiring target address information and processing rules according to the analysis result;
respectively establishing a transmission link with the source database and a transmission link with the target database according to the target address information;
and determining the target data through the processing rule and the identification information of the target data.
6. A data synchronization system, comprising: a source database, a target database and a server,
the server is respectively connected with the source database and the target database;
the server is configured to obtain a target configuration file, where the target configuration file includes: configuration information of the source database, configuration information of the target database and identification information of the target data;
generating target request information according to the target configuration file, wherein the request information is used for requesting the source database to send target data;
sending the target request information to the source database; and obtaining a feedback result according to the target request information, and sending the feedback result to the target database.
The source database is used for acquiring target data according to the target request data and sending the target data to the server;
and the target database is used for receiving the feedback result.
7. The system of claim 6, wherein the source database is further configured to:
determining whether the source database can find the target data or not according to the target request information;
when the target data are found, generating a feedback result according to the target data;
when the target data is not found, generating target information, and generating a feedback result according to the target information, wherein the target information is used for indicating that the target data is not obtained from the source database;
and sending the feedback result to the server.
8. A server, characterized in that the server comprises: a transmitter, a processor and a receiver, wherein,
the processor is respectively connected with the transmitter and the receiver;
the receiver is configured to obtain a target profile, where the target profile includes: configuration information of the source database, configuration information of the target database and identification information of the target data;
the processor is used for generating target request information according to the target configuration file and sending the target request information to the source database, wherein the request information is used for requesting the source database to send target data;
and the transmitter is used for acquiring a feedback result according to the target request information and transmitting the feedback result to the target database.
9. The server of claim 8, wherein the processor is further configured to,
when the feedback result shows that the target data is successfully acquired, acquiring the target data;
and converting the target data according to a preset data format, and sending the converted target data to the target database.
10. The server of claim 8, wherein the processor is further configured to,
analyzing the target configuration file, and acquiring target address information and processing rules according to the analysis result;
respectively establishing a transmission link with the source database and a transmission link with the target database according to the target address information;
and determining the target data through the processing rule and the identification information of the target data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911356636.9A CN113032480A (en) | 2019-12-25 | 2019-12-25 | Data synchronization method, system and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911356636.9A CN113032480A (en) | 2019-12-25 | 2019-12-25 | Data synchronization method, system and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113032480A true CN113032480A (en) | 2021-06-25 |
Family
ID=76458190
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911356636.9A Pending CN113032480A (en) | 2019-12-25 | 2019-12-25 | Data synchronization method, system and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032480A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2891994A1 (en) * | 2013-11-04 | 2015-07-08 | Guangdong Electronics Industry Institute Ltd. | Method for achieving automatic synchronization of multisource heterogeneous data resources |
| CN105787077A (en) * | 2016-03-02 | 2016-07-20 | 百度在线网络技术（北京）有限公司 | Data synchronizing method and device |
| CN106469158A (en) * | 2015-08-17 | 2017-03-01 | 杭州海康威视系统技术有限公司 | Method of data synchronization and device |
| CN108228814A (en) * | 2017-12-29 | 2018-06-29 | 泰康保险集团股份有限公司 | Method of data synchronization and device |
| CN109753492A (en) * | 2019-01-02 | 2019-05-14 | 陕西西部资信股份有限公司 | A kind of system constituting method and device based on metadata configurations |
| CN110147411A (en) * | 2019-05-20 | 2019-08-20 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
-
2019
- 2019-12-25 CN CN201911356636.9A patent/CN113032480A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2891994A1 (en) * | 2013-11-04 | 2015-07-08 | Guangdong Electronics Industry Institute Ltd. | Method for achieving automatic synchronization of multisource heterogeneous data resources |
| CN106469158A (en) * | 2015-08-17 | 2017-03-01 | 杭州海康威视系统技术有限公司 | Method of data synchronization and device |
| CN105787077A (en) * | 2016-03-02 | 2016-07-20 | 百度在线网络技术（北京）有限公司 | Data synchronizing method and device |
| CN108228814A (en) * | 2017-12-29 | 2018-06-29 | 泰康保险集团股份有限公司 | Method of data synchronization and device |
| CN109753492A (en) * | 2019-01-02 | 2019-05-14 | 陕西西部资信股份有限公司 | A kind of system constituting method and device based on metadata configurations |
| CN110147411A (en) * | 2019-05-20 | 2019-08-20 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110032459B (en) | Message queue configuration method and device, computer equipment and storage medium | |
| CN109582588B (en) | Test case generation method and device and electronic equipment | |
| CN106850830B (en) | Service request processing method, device and system and related server | |
| CN102637214B (en) | Method and system for synchronizing general data among database services | |
| EP3214559A1 (en) | Data graphing method, device and database server | |
| CN105548749B (en) | A kind of communication terminal test method, apparatus Internet-based and system | |
| CN111625585B (en) | Access method, device, host and storage medium of hardware acceleration database | |
| CN110134583B (en) | Software testing and data processing method and device | |
| US20190108163A1 (en) | Database expansion system, equipment, and method of expanding database | |
| CN107277087B (en) | Data processing method and device | |
| US10171606B2 (en) | System and method for providing data as a service (DaaS) in real-time | |
| CN110737655B (en) | Method and device for reporting data | |
| CN115495489A (en) | Cross-border logistics order track query method and device, terminal equipment and storage medium | |
| CN110943876B (en) | URL state detection method, device, equipment and system | |
| CN111339047B (en) | Data interaction method, device and system | |
| CN113032480A (en) | Data synchronization method, system and device | |
| US20100198947A1 (en) | System and Method for Dynamically Processing Electronic Data Between Multiple Data Sources | |
| CN106649678B (en) | Data processing method and system | |
| CN110737662A (en) | data analysis method, device, server and computer storage medium | |
| CN112422314B (en) | Method and system for collecting and reporting alarm performance data | |
| CN112948195B (en) | Interface testing method, device, electronic equipment and storage medium | |
| CN113434513A (en) | User tag data storage method, device, system, equipment and storage medium | |
| CN110740046B (en) | Method and device for analyzing service contract | |
| CN113076343A (en) | Data query method, device, equipment and storage medium | |
| CN109086145B (en) | Data generation method and device and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210625 |
|
| RJ01 | Rejection of invention patent application after publication |