CN111367883A - Data synchronization method, device, equipment and computer readable storage medium - Google Patents
Data synchronization method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111367883A CN111367883A CN202010115404.0A CN202010115404A CN111367883A CN 111367883 A CN111367883 A CN 111367883A CN 202010115404 A CN202010115404 A CN 202010115404A CN 111367883 A CN111367883 A CN 111367883A
- Authority
- CN
- China
- Prior art keywords
- data
- source
- data synchronization
- script
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Abstract
The invention relates to the technical field of big data, and discloses a data synchronization method, which comprises the following steps: acquiring a data synchronization request; acquiring source data to be synchronized in the source data source and parameter information of a target data source according to the data synchronization request; generating a data synchronization script based on the service logic information, the parameter information and a preset universal script template, wherein the universal script template is obtained by constructing script code public parameters based on extracted synchronous data among different data sources; synchronizing source data on the source data source to the target data source based on the data synchronization script. The invention also discloses a data synchronization device, equipment and a computer readable storage medium. According to the invention, the data synchronization script is generated by the preset universal script template, so that the generation efficiency of the data synchronization script is improved, and further, the data synchronization efficiency is improved.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a data synchronization method, apparatus, device, and computer-readable storage medium.
Background
At present, with the development and convenience of the mobile internet, more and more enterprises are in data-oriented operation, so that a source data source is continuously expanded, and may include structured databases such as ORACLE, MYSQL, and the like, semi-structured databases such as MON GODB, post, high, and the like, it is important how to efficiently acquire source data from each database of the data source, and also synchronize the source data to a specified target data source. Therefore, how to improve the efficiency of data synchronization is a technical problem to be solved urgently at present.
Disclosure of Invention
The invention mainly aims to provide a data synchronization method, a data synchronization device, data synchronization equipment and a computer readable storage medium, and aims to solve the technical problem of low data synchronization efficiency.
In order to achieve the above object, the present invention provides a data synchronization method, including the steps of:
acquiring a data synchronization request, wherein the data synchronization request comprises source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source, and service logic information for realizing data synchronization between the source data source and the target data source;
analyzing the data synchronization request to obtain an analysis result;
generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting common parameters in script codes for realizing data synchronization among different data sources;
synchronizing source data on the source data source to the target data source based on the data synchronization script.
Optionally, before the step of synchronizing the source data on the source data source to the target data source based on the data synchronization script, the method further includes the following steps:
sending the data synchronization script to a preset safe storage device for storage and backup;
reading a storage path of the data synchronization script in the secure storage device, and generating an execution path;
the step of synchronizing source data on the source data source to the target data source based on the data synchronization script comprises:
and according to the execution path, executing the data synchronization script stored in the secure storage through a mirror image mapping operation principle, and synchronizing the source data on the source data source to the target data source.
Optionally, if the preset secure storage device is a distributed file system, the sending the data synchronization script to the preset secure storage device for storage and backup includes the following steps:
sending the data synchronization script to the distributed file system, wherein the distributed file system comprises a main node and a plurality of slave nodes, the main node is used for storing metadata of the data synchronization script, and the slave nodes are used for storing the data synchronization script;
storing and backing up the data synchronization script by the slave node based on the metadata. Optionally, the generating a data synchronization script based on the parsing result and a preset universal script template, where the universal script template is constructed by extracting common parameters from script codes for implementing data synchronization between different data sources, includes the following steps:
acquiring version information of a data synchronization script to be generated based on the analysis result;
selecting a function from a preset function database based on the version information, and calling the function to generate a preset universal script template;
and generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting common parameters in script codes for realizing data synchronization among different data sources.
Optionally, the executing the data synchronization script stored in the secure storage according to the execution path by the principle of mirror mapping operation, and synchronizing the source data on the source data source to the target data source includes the following steps:
encrypting the account name and the password of the source data source into a ciphertext through a secret key;
the data synchronization request further comprises a ciphertext of the source data source;
decrypting the ciphertext to obtain a decrypted account name and a decrypted password;
acquiring source data from the source data source according to the decrypted account name and the decrypted password;
and executing the data synchronization script according to the execution path, executing the data synchronization script stored in the safe storage through a mirror image mapping operation principle, and synchronizing the source data to the target data source.
Optionally, the selecting a function from a preset function database based on the version information, and calling the function to generate a data synchronization script includes the following steps:
pre-establishing a mapping relation between preset version information and a function of a preset function database;
and selecting a function from a preset function database according to the version information based on the mapping relation, and calling the function to generate a data synchronization script.
Optionally, before the step of synchronizing the source data on the source data source to the target data source based on the data synchronization script, the method further includes the following steps:
detecting whether source data to be synchronized are synchronized to a preset target data source from a preset source data source;
if the source data to be synchronized is synchronized to a preset target data source from a preset source data source, outputting prompt information that the data synchronization script runs successfully;
and if the source data to be synchronized is not synchronized to the preset target data source from the preset source data source, outputting prompt information of the operation failure of the data synchronization script.
Further, to achieve the above object, the present invention provides a data synchronization apparatus, including the following modules:
the system comprises a receiving module, a synchronization module and a processing module, wherein the receiving module is used for acquiring a data synchronization request, and the data synchronization request comprises source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source and service logic information for realizing data synchronization between the source data source and the target data source;
the analysis module is used for analyzing the data synchronization request to obtain an analysis result;
the script generation module is used for generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting public parameters in script codes for realizing data synchronization among different data sources;
and the data synchronization module is used for synchronizing the source data on the source data source to the target data source based on the data synchronization script.
Optionally, the data synchronization apparatus further includes the following modules:
the storage and backup module is used for sending the data synchronization script to a preset safe storage device for storage and backup;
the execution path generation module is used for reading a storage path of the data synchronization script in the safe storage equipment and generating an execution path;
and the data synchronization script execution module is used for executing the data synchronization script stored in the secure storage according to the execution path by a mirror image mapping operation principle, and synchronizing the source data on the source data source to the target data source.
Optionally, the storage and backup module comprises the following units:
the data synchronization script sending unit is used for sending the data synchronization script to the distributed file system, the distributed file system comprises a main node and a plurality of slave nodes, the main node is used for storing metadata of the data synchronization script, and the slave nodes are used for storing the data synchronization script;
and the data synchronization script storage and backup unit is used for storing and backing up the data synchronization script through the slave node based on the metadata.
Optionally, the script generation module includes the following units:
the version information acquisition unit is used for acquiring the version information of the data synchronization script to be generated based on the analysis result;
a function calling unit for selecting a function from a preset function database based on the version information and calling the function to generate a preset universal script template;
and the script generating unit is used for generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting public parameters in script codes for realizing data synchronization among different data sources.
Optionally, the data synchronization script execution module includes the following units:
the encryption unit is used for encrypting the account name and the password of the source data source into a ciphertext through a secret key; the data synchronization request further comprises a ciphertext of the source data source;
the decryption unit is used for decrypting the ciphertext to obtain a decrypted account name and a decrypted password;
a source data obtaining unit, configured to obtain source data from the source data source according to the decrypted account name and password;
and the source data synchronization unit is used for executing the data synchronization script according to the execution path, executing the data synchronization script stored in the safe storage through a mirror image mapping operation principle, and synchronizing the source data to the target data source.
Optionally, the calling function unit is configured to:
pre-establishing a mapping relation between preset version information and a function of a preset function database;
and selecting a function from a preset function database according to the version information based on the mapping relation, and calling the function to generate a data synchronization script.
Optionally, the data synchronization apparatus further includes the following modules:
the device comprises a detection module, a synchronization module and a synchronization module, wherein the detection module is used for detecting whether source data to be synchronized are synchronized to a preset target data source from a preset source data source;
the operation success prompting module is used for outputting the prompt information that the data synchronization script is successfully operated if the source data to be synchronized is synchronized from a preset source data source to a preset target data source;
and the operation failure prompt module is used for outputting prompt information of the operation failure of the data synchronization script if the source data to be synchronized is not synchronized to the preset target data source from the preset source data source.
Further, to achieve the above object, the present invention also provides a data synchronization device, which includes a memory, a processor, and a data synchronization program stored on the memory and operable on the processor, wherein the data synchronization program, when executed by the processor, implements the steps of the data synchronization method according to any one of the above.
Further, to achieve the above object, the present invention also provides a computer readable storage medium having stored thereon a data synchronization program, which when executed by a processor, implements the steps of the data synchronization method according to any one of the above.
The invention acquires the user configuration information, inputs the configuration information into the preset universal script template, and generates the data synchronization script through the preset universal script template, thereby avoiding low data synchronization efficiency caused by constructing the template for many times. The invention can greatly reduce the difficulty of script generation, simplify the flow of template generation and improve the efficiency of data synchronization.
Drawings
Fig. 1 is a schematic structural diagram of an operating environment of a data synchronization device according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a data synchronization method according to a first embodiment of the present invention;
FIG. 3 is a detailed flowchart of one embodiment of step S40 in FIG. 2;
FIG. 4 is a detailed flowchart of one embodiment of step S40 in FIG. 2;
FIG. 5 is a detailed flowchart of one embodiment of step S401 in FIG. 3;
FIG. 6 is a detailed flowchart of one embodiment of step S30 in FIG. 2;
FIG. 7 is a detailed flowchart of one embodiment of step S50 in FIG. 4;
FIG. 8 is a detailed flowchart of one embodiment of step S302 in FIG. 6;
FIG. 9 is a flowchart illustrating a data synchronization method according to a second embodiment of the present invention;
fig. 10 is a functional block diagram of an embodiment of a data synchronization apparatus according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a data synchronization device.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an operating environment of a data synchronization device according to an embodiment of the present invention.
As shown in fig. 1, the data synchronization apparatus includes: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display (Display), an input unit such as a Keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the hardware configuration of the data synchronization device shown in fig. 1 does not constitute a limitation of the data synchronization device, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer-readable storage medium, may include therein an operating system, a network communication module, a user interface module, and a data synchronization program. Among them, the operating system is a program that manages and controls the data synchronization apparatus and software resources, supporting the operation of the data synchronization program as well as other software and/or programs.
In the hardware configuration of the data synchronization apparatus shown in fig. 1, the network interface 1004 is mainly used for accessing a network; the user interface 1003 is mainly used for detecting a confirmation instruction, an editing instruction, and the like. And the processor 1001 may be configured to invoke a data synchronization program stored in the memory 1005 and perform the operations of the various embodiments of the data synchronization method below.
Based on the hardware structure of the data synchronization device, the embodiments of the data synchronization method are provided.
Referring to fig. 2, fig. 2 is a flowchart illustrating a data synchronization method according to a first embodiment of the present invention. In this embodiment, the data synchronization method includes the following steps:
step S10, acquiring a data synchronization request, wherein the data synchronization request includes source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source, and service logic information for realizing data synchronization between the source data source and the target data source;
in this embodiment, the setting of each parameter in the request may be specifically performed on a data synchronization platform orThe man-machine interaction interface provided on the terminal is set by the user, after the setting is completed, the system or the terminal generates a corresponding instruction according to the setting on the interface, namely an operation instruction for data synchronization received by the man-machine interaction interface, and the terminal equipment of the man-machine interaction interface can be a smart phone, a tablet electronic equipment, a portable computer, a personal digital assistant, a personal computer or an all-in-one machine and the like. For example, receiving a synchronization of data in a MYSQL database to Hive1The instruction of (1). After the machine receives the operation instruction, a task instruction is generated according to the operation instruction, the task instruction comprises content information of the task to be executed, and the content information comprises the type of the task to be executed.
Step S20, analyzing the data synchronization request to obtain an analysis result;
in this embodiment, after receiving the data synchronization request, the data synchronization request is analyzed, so that the parameter information of the source data source and the information of the target number source can be obtained. For example, after receiving the current data synchronization request, the current data synchronization request is parsed, so as to obtain parameter information and Hive of the MYSQL database1The parameter information of the target data source and the source data to be synchronized in the source data source are obtained, and the source data to be synchronized can be synchronized from the source data source to the target data source conveniently.
Step S30, generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting public parameters in script codes for realizing data synchronization among different data sources;
in this embodiment, a common code is extracted from script codes for implementing data synchronization between different data sources in advance, and a preset universal script template is generated based on the common code, because the data synchronization request includes source data information to be synchronized in the source data source, parameter information of the target data source, and service logic information for implementing data synchronization between the source data source and the target data source, an analysis result obtained after the analysis also includes the service logic information for synchronizing the data to be synchronized from the source data source to the target data source, and a data synchronization script can be generated according to the analysis result and the preset universal script template.
Compared with the prior art, one script is needed for each data synchronization, especially in a big data scene, the source data can come from different source data sources, so that a plurality of data synchronization scripts corresponding to the different source data sources need to be prepared in advance, when the offline data are synchronized, the data are synchronized from the source data sources to the target data sources by providing a plurality of interfaces, even a manual script writing mode is used, one script is written for one data or a batch of data, and the data synchronization script is generated through a preset universal script template, so that the efficiency of generating the data synchronization script can be improved.
Step S40, synchronizing the source data on the source data source to the target data source based on the data synchronization script.
In this embodiment, a data synchronization script is executed to synchronize data from a source data source to a target data source, where the source data source includes structured databases such as ORACLE, MYSQL, etc., and semi-structured databases such as MONGODB, postcred, HIVE, etc. Because the data synchronization script is generated through the preset universal script template, the generation flow of the data synchronization script is simplified, the difficulty of script generation is reduced, the script generation efficiency is improved, and the data synchronization efficiency is further improved.
Referring to fig. 3, fig. 3 is a detailed flowchart of an embodiment of step S40 in fig. 2. In this embodiment, step S40, synchronizing the source data on the source data source to the target data source based on the data synchronization script, includes the following steps:
step S401, sending the data synchronization script to a preset safe storage device for storage and backup;
in this embodiment, the secure storage device may be a distributed file system, that is, an HDFS, configured to access data, and the HDFS is a master-slave structure and is composed of a master node and a slave node. The main node is used for receiving user operation requests, maintaining a directory structure of the file system and managing the relationship between files and blocks. The slave nodes are mainly used for storing files, and the files are stored on a disk in blocks.
Step S402, reading the storage path of the data synchronization script in the safe storage device, and generating an execution path.
In this embodiment, after parsing the data synchronization request, source data source information, target data source information, and service logic information for implementing data synchronization between the source data source and the target data source are obtained, a synchronization script storage path is predetermined and stored in a secure storage device, and an execution path of a script is obtained according to the storage path, for example, the current HDFS has a master node and three slave nodes, where the three slave nodes are a first slave node, a second slave node, and a third slave node, and for example, the data synchronization path may be a path in which after receiving an instruction for synchronizing data from the master node, data is sequentially synchronized to the first slave node, then synchronized to the second slave node, then synchronized to the third slave node, and finally synchronized to the target data source.
Different types of data refer to data from different source data sources, e.g., structured databases from ORACLE, MYSQL, etc. After receiving the instruction of synchronizing the data, the data synchronization script is executed according to the execution path, so that the data is synchronized from the source data source to the target data source, for example, MYSQL is synchronized to different HIVEs.
Referring to fig. 4, fig. 4 is a detailed flowchart of an embodiment of step S40 in fig. 2. In this embodiment, in step S40, the step of synchronizing the source data on the source data source to the target data source based on the data synchronization script includes:
and step S50, according to the execution path, executing the data synchronization script stored in the secure storage through the principle of mirror image mapping operation, and synchronizing the source data on the source data source to the target data source.
In the embodiment, the execution path is mirrored and mapped to the preset execution module, so that the execution of the data synchronization script is realized according to the execution path.
Referring to fig. 5, fig. 5 is a detailed flowchart of an embodiment of step S401 in fig. 3. In this embodiment, when the following conditions exist in step S401, and if the preset secure storage device is a distributed file system, the method sends the data synchronization script to the preset secure storage device for storage and backup, and further includes the following steps:
step S4011, sending the data synchronization script to a distributed file system, wherein the distributed file system comprises a master node and a plurality of slave nodes, the master node is used for storing metadata of the data synchronization script, and the slave nodes are used for storing the data synchronization script;
in this embodiment, metadata is used to indicate the storage location of the data synchronization script. For example, whether it is stored in a first slave node or a second slave node. When the data stored in each slave node is changed, the master node receives the changed message and changes the information of the metadata. The data synchronization script is independently stored in the slave node, so that the data synchronization script and the data synchronization script are independent from each other, and the risk of tampering the history script is reduced.
Step S4012, storing and backing up the data synchronization script through the slave node based on the metadata.
In this embodiment, a heartbeat packet is used to send a message of a slave node within a preset time to a master node, for example, a heartbeat is sent every 5 seconds, so that the status of each slave node can be monitored, when a certain slave node fails, the failed message can be sent to the master node, the master node backs up data stored in the failed slave node to other slave nodes according to metadata information, and the distributed file system has an automatic backup mechanism, so that the stability of each data synchronization script can be ensured. For example, when a first slave node fails, a message of the failure may be sent to the master node, and the master node backs up data stored in the first slave node to a second slave node according to the metadata information.
Referring to fig. 6, fig. 6 is a detailed flowchart of an embodiment of step S30 in fig. 2. In this embodiment, in step S30, a data synchronization script is generated based on the parsing result and a preset universal script template, where the universal script template is constructed by extracting common parameters from script codes for implementing data synchronization between different data sources, and the method includes the following steps:
step S301, acquiring version information of a data synchronization script to be generated based on an analysis result;
in this embodiment, whether the human-computer interaction interface triggers an instruction is monitored in real time to determine whether a preparation stage for generating the data synchronization script is entered, and when it is monitored that the human-computer interaction interface has the triggering instruction, the version information of the data synchronization script is obtained according to a preset mapping relationship between an analysis result and the version information of the data synchronization script.
Step S302, selecting a function from a preset function database based on version information, and calling the function to generate a preset universal script template;
step S303, generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is constructed by extracting common parameters in script codes for realizing data synchronization among different data sources.
In this embodiment, a preset mapping relationship exists between the version information and the function in the preset function database, so that the function can be selected from the preset function database based on the version information, the service logic for generating the data synchronization script is realized according to the function, and the data synchronization script is obtained.
Referring to fig. 7, fig. 7 is a detailed flowchart of an embodiment of step S50 in fig. 4. In this embodiment, step S50, according to the execution path, executes the data synchronization script stored in the secure storage according to the principle of mirror mapping operation, and synchronizes the source data on the source data source to the target data source, including the following steps:
step S501, encrypting an account name and a password of a source data source into a ciphertext through a secret key;
the data synchronization request also comprises a ciphertext of the source data source;
in this embodiment, in order to ensure the security of the source data, the account name and the password for storing the source data are encrypted, and only after the source data are successfully decrypted, the source data in the source data source can be acquired, and the password is stored in the Hadoop, which is an open-source distributed parallel processing platform.
Step S502, the ciphertext is decrypted to obtain the decrypted account name and the decrypted password;
in this embodiment, since operations such as decryption and data synchronization are triggered only when a data synchronization request is received, the data synchronization request needs to be obtained first, and whether the data synchronization request is received can be monitored by a preset monitoring program.
The user name and the password can be encrypted according to a preset encryption algorithm, and only after the encrypted ciphertext is successfully decrypted, the source data of the source data source can be obtained.
Step S503, acquiring source data from a source data source according to the decrypted account name and the password;
in this embodiment, each account name and password corresponds to different source data in the source data source, so that the source data can be selectively acquired according to the account name and the password.
Step S504, executing the data synchronization script according to the execution path, executing the data synchronization script stored in the secure storage according to the mirror mapping operation principle, and synchronizing the source data to the target data source.
In this embodiment, only after decryption is successful, the source data in the source data source may be acquired, and therefore, an execution path needs to be established in advance, and when synchronizing data, a data synchronization script stored in a secure storage is executed according to a mirror mapping operation principle, and finally, the source data is synchronized to the target data source through different HDFS paths.
Referring to fig. 8, fig. 8 is a schematic view of a detailed flow of an embodiment of step S302 in fig. 6. In this embodiment, step S302, based on the version information, selects a function from a preset function database, and calls the function to generate a data synchronization script, including the following steps:
step S3021, pre-establishing a mapping relationship between the preset version information and a function of the preset function database;
in this embodiment, when a data synchronization script is generated through a preset universal script template, data synchronization scripts of different versions need to be generated according to a data synchronization request, and since different functions need to be called to execute a data synchronization script instruction for generating a corresponding version when a script is generated, a corresponding relationship between version information and a function to be called needs to be established in advance, so that a corresponding function can be acquired according to known version information. Specifically, the mapping relationship may be established through a hash table.
And step S3022, selecting a function from the preset function database according to the version information based on the mapping relation, and calling the function to generate a data synchronization script.
In this embodiment, the different source data may be from different source data sources, for example, structured databases such as ORACLE, MYSQL, etc., semi-structured databases such as MON GODB, postdred, HIVE, etc. Because the mapping relation exists between the version information and the function to be called, the specified function can be called, and the business logic of generating the data synchronization script is realized. The information of the business logic includes addition, reading, updating and deleting operations of data. For example, when a data synchronization a to a database b is currently received, corresponding access paths of a source data source and a target data source where the data a is located are obtained, information of the source data source and the target data source and respective access paths are added to a service logic for generating a data synchronization script, and a corresponding synchronization script is generated.
Referring to fig. 9, fig. 9 is a flowchart illustrating a data synchronization method according to a second embodiment of the present invention. In this embodiment, in step S40 of fig. 2, before synchronizing the source data on the source data source to the target data source based on the data synchronization script, the method further includes the following steps:
step S60, detecting whether the source data to be synchronized is synchronized from the preset source data source to the preset target data source;
in the present embodiment, after the data synchronization script is generated, in order to ensure that the data to be synchronized can be synchronized from the preset source data source to the preset target data source when the data synchronization script is executed, a pre-commissioning is required.
Step S70, if the source data to be synchronized is synchronized from the preset source data source to the preset target data source, outputting a prompt message that the data synchronization script runs successfully;
in this embodiment, whether each script in the data synchronization script can be accurately run is detected by detecting whether the data to be synchronized is synchronized from the preset source data source to the preset target data source.
In this embodiment, if the data to be synchronized is synchronized from the preset source data source to the preset target data source, a prompt message indicating that the data synchronization script runs successfully is output, which indicates that each data synchronization script in the data synchronization script can implement the task of data synchronization, and therefore, the data synchronization can be implemented by executing the script.
Step S80, if the source data to be synchronized is not synchronized from the preset source data source to the preset target data source, outputting a prompt message indicating that the data synchronization script fails to run.
In this embodiment, if the data to be synchronized is not synchronized from the preset source data source to the preset target data source, a prompt message indicating that the data synchronization script fails to run is output. And the data synchronization script with problems can be effectively screened out through advanced detection.
Referring to fig. 10, fig. 10 is a functional block diagram of an embodiment of a data synchronization apparatus according to the present invention. In this embodiment, the data synchronization apparatus includes the following modules:
a receiving module 10, configured to obtain a data synchronization request, where the data synchronization request includes source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source, and service logic information for implementing data synchronization between the source data source and the target data source;
the analysis module 20 is used for analyzing the data synchronization request to obtain an analysis result;
the script generation module 30 is configured to generate a data synchronization script based on the parsing result and a preset universal script template, where the universal script template is constructed by extracting common parameters from script codes for implementing data synchronization between different data sources;
and the data synchronization module 40 is used for synchronizing the source data on the source data source to the target data source based on the data synchronization script.
In this embodiment, the receiving module 10 is configured to obtain a data synchronization request, where the data synchronization request includes source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source, and service logic information for implementing data synchronization between the source data source and the target data source; the analysis module 20 is configured to analyze the data synchronization request to obtain an analysis result; the script generation module 30 is configured to generate a data synchronization script based on the parsing result and a preset universal script template, where the universal script template is constructed by extracting common parameters from script codes for implementing data synchronization between different data sources; the data synchronization module 40 is configured to synchronize source data on a source data source to a target data source based on a data synchronization script. The module of the data synchronization device can realize that the corresponding data synchronization script is generated through the preset universal script template, thereby improving the generation efficiency of the data synchronization script and further improving the efficiency of data synchronization.
The invention also provides a computer readable storage medium.
In this embodiment, the computer readable storage medium has a data synchronization program stored thereon, and the data synchronization program, when executed by a processor, implements the steps of the data synchronization method as described in any one of the above embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM), and includes instructions for enabling a terminal (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The present invention is described in connection with the accompanying drawings, but the present invention is not limited to the above embodiments, which are only illustrative and not restrictive, and those skilled in the art can make various changes without departing from the spirit and scope of the invention as defined by the appended claims, and all changes that come within the meaning and range of equivalency of the specification and drawings that are obvious from the description and the attached claims are intended to be embraced therein.
Claims (10)
1. A data synchronization method, characterized in that the data synchronization method comprises the steps of:
acquiring a data synchronization request, wherein the data synchronization request comprises source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source, and service logic information for realizing data synchronization between the source data source and the target data source;
analyzing the data synchronization request to obtain an analysis result;
generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting common parameters in script codes for realizing data synchronization among different data sources;
synchronizing source data on the source data source to the target data source based on the data synchronization script.
2. The data synchronization method of claim 1, further comprising, prior to the step of synchronizing source data on the source data source to the target data source based on the data synchronization script, the steps of:
sending the data synchronization script to a preset safe storage device for storage and backup;
reading a storage path of the data synchronization script in the secure storage device, and generating an execution path;
the step of synchronizing source data on the source data source to the target data source based on the data synchronization script comprises:
and according to the execution path, executing the data synchronization script stored in the secure storage through a mirror image mapping operation principle, and synchronizing the source data on the source data source to the target data source.
3. The data synchronization method of claim 2, wherein if the preset secure storage device is a distributed file system, the step of sending the data synchronization script to the preset secure storage device for storage and backup comprises the following steps:
sending the data synchronization script to the distributed file system, wherein the distributed file system comprises a main node and a plurality of slave nodes, the main node is used for storing metadata of the data synchronization script, and the slave nodes are used for storing the data synchronization script;
storing and backing up the data synchronization script by the slave node based on the metadata.
4. The data synchronization method according to claim 1, wherein the data synchronization script is generated based on the parsing result and a preset universal script template, the universal script template is constructed by extracting common parameters in script codes for realizing data synchronization between different data sources, and the method comprises the following steps:
acquiring version information of a data synchronization script to be generated based on the analysis result;
selecting a function from a preset function database based on the version information, and calling the function to generate a preset universal script template;
and generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting common parameters in script codes for realizing data synchronization among different data sources.
5. The data synchronization method of claim 2, wherein the executing the data synchronization script stored in the secure storage according to the execution path by the principle of mirror mapping operation, the synchronizing the source data on the source data source to the target data source comprises the steps of:
encrypting the account name and the password of the source data source into a ciphertext through a secret key;
the data synchronization request further comprises a ciphertext of the source data source;
decrypting the ciphertext to obtain a decrypted account name and a decrypted password;
acquiring source data from the source data source according to the decrypted account name and the decrypted password;
and executing the data synchronization script according to the execution path, executing the data synchronization script stored in the safe storage through a mirror image mapping operation principle, and synchronizing the source data to the target data source.
6. The data synchronization method of claim 4, wherein the selecting a function from a preset function database based on the version information, and calling the function to generate a data synchronization script comprises the steps of:
pre-establishing a mapping relation between preset version information and a function of a preset function database;
and selecting a function from a preset function database according to the version information based on the mapping relation, and calling the function to generate a data synchronization script.
7. The data synchronization method of claim 1, further comprising, prior to the step of synchronizing source data on the source data source to the target data source based on the data synchronization script, the steps of:
detecting whether source data to be synchronized are synchronized to a preset target data source from a preset source data source;
if the source data to be synchronized is synchronized to a preset target data source from a preset source data source, outputting prompt information that the data synchronization script runs successfully;
and if the source data to be synchronized is not synchronized to the preset target data source from the preset source data source, outputting prompt information of the operation failure of the data synchronization script.
8. A data synchronization apparatus, comprising:
the system comprises a receiving module, a synchronization module and a processing module, wherein the receiving module is used for acquiring a data synchronization request, and the data synchronization request comprises source data information to be synchronized in a source data source, parameter information of the source data source, parameter information of a target data source and service logic information for realizing data synchronization between the source data source and the target data source;
the analysis module is used for analyzing the data synchronization request to obtain an analysis result;
the script generation module is used for generating a data synchronization script based on the analysis result and a preset universal script template, wherein the universal script template is obtained by extracting public parameters in script codes for realizing data synchronization among different data sources;
and the data synchronization module is used for synchronizing the source data on the source data source to the target data source based on the data synchronization script.
9. A data synchronization device, characterized in that the data synchronization device comprises a memory, a processor and a data synchronization program stored on the memory and executable on the processor, which data synchronization program, when executed by the processor, implements the steps of the data synchronization method according to any of claims 1-7.
10. A computer-readable storage medium, having stored thereon a data synchronization program, which when executed by a processor implements the steps of the data synchronization method of any one of claims 1-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010115404.0A CN111367883A (en) | 2020-02-25 | 2020-02-25 | Data synchronization method, device, equipment and computer readable storage medium |
| PCT/CN2020/118115 WO2021169277A1 (en) | 2020-02-25 | 2020-09-27 | Data synchronization method, apparatus and device, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010115404.0A CN111367883A (en) | 2020-02-25 | 2020-02-25 | Data synchronization method, device, equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111367883A true CN111367883A (en) | 2020-07-03 |
Family
ID=71210383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010115404.0A Pending CN111367883A (en) | 2020-02-25 | 2020-02-25 | Data synchronization method, device, equipment and computer readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111367883A (en) |
| WO (1) | WO2021169277A1 (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111984730A (en) * | 2020-08-24 | 2020-11-24 | 中电九天智能科技有限公司 | Centralized data synchronization method for industrial internet |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112347080A (en) * | 2020-11-11 | 2021-02-09 | 金蝶云科技有限公司 | Data migration method and related device |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN112364049A (en) * | 2020-11-10 | 2021-02-12 | 中国平安人寿保险股份有限公司 | Data synchronization script generation method, system, terminal and storage medium |
| CN112612775A (en) * | 2020-12-17 | 2021-04-06 | 平安消费金融有限公司 | Data storage method and device, computer equipment and storage medium |
| WO2021169277A1 (en) * | 2020-02-25 | 2021-09-02 | 平安科技（深圳）有限公司 | Data synchronization method, apparatus and device, and computer-readable storage medium |
| CN113448775A (en) * | 2021-06-25 | 2021-09-28 | 中国工商银行股份有限公司 | Multi-source heterogeneous data backup method and device |
| WO2022126918A1 (en) * | 2020-12-17 | 2022-06-23 | 平安科技（深圳）有限公司 | Method and apparatus for automatically synchronizing associated scripts, and computer device and storage medium |
| CN114928620A (en) * | 2022-05-31 | 2022-08-19 | 曙光信息产业股份有限公司 | User information synchronization method, device, equipment, storage medium and program product |
| CN115033647A (en) * | 2022-08-11 | 2022-09-09 | 杭州湖畔网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN116204509A (en) * | 2023-02-14 | 2023-06-02 | 中电金信软件有限公司 | Data access script generation method and device, electronic equipment and storage medium |
| CN117077218A (en) * | 2023-10-07 | 2023-11-17 | 之江实验室 | Synchronization method, device, medium and equipment for mirror images of multi-center medical institution |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113821565B (en) * | 2021-09-10 | 2024-03-15 | 上海得帆信息技术有限公司 | Method for synchronizing data by multiple data sources |
| CN114840598B (en) * | 2022-07-04 | 2022-10-04 | 好享家舒适智能家居股份有限公司 | Method and system for creating data source in runtime and automatically switching |
| CN116541019B (en) * | 2023-06-28 | 2023-09-29 | 深圳市极限网络科技有限公司 | Data forwarding method, system, device and readable storage medium |
| CN116560697B (en) * | 2023-07-06 | 2023-09-15 | 上海鉴智其迹科技有限公司 | DBC database synchronization method, device, electronic equipment and readable storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7103844B2 (en) * | 2002-06-26 | 2006-09-05 | International Business Machines Corporation | Portal/portlet application data synchronization |
| CN107908631B (en) * | 2017-07-25 | 2021-04-20 | 平安科技（深圳）有限公司 | Data processing method, data processing device, storage medium and computer equipment |
| CN107967316A (en) * | 2017-11-22 | 2018-04-27 | 平安科技（深圳）有限公司 | A kind of method of data synchronization, equipment and computer-readable recording medium |
| CN109271447A (en) * | 2018-09-04 | 2019-01-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111367883A (en) * | 2020-02-25 | 2020-07-03 | 平安科技（深圳）有限公司 | Data synchronization method, device, equipment and computer readable storage medium |
-
2020
- 2020-02-25 CN CN202010115404.0A patent/CN111367883A/en active Pending
- 2020-09-27 WO PCT/CN2020/118115 patent/WO2021169277A1/en active Application Filing
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021169277A1 (en) * | 2020-02-25 | 2021-09-02 | 平安科技（深圳）有限公司 | Data synchronization method, apparatus and device, and computer-readable storage medium |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112287021B (en) * | 2020-07-13 | 2024-04-05 | 上海柯林布瑞信息技术有限公司 | Method and device for generating and synchronizing real-time data synchronization parameters, storage medium and terminal |
| CN111984730A (en) * | 2020-08-24 | 2020-11-24 | 中电九天智能科技有限公司 | Centralized data synchronization method for industrial internet |
| CN112364049A (en) * | 2020-11-10 | 2021-02-12 | 中国平安人寿保险股份有限公司 | Data synchronization script generation method, system, terminal and storage medium |
| CN112347080A (en) * | 2020-11-11 | 2021-02-09 | 金蝶云科技有限公司 | Data migration method and related device |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| WO2022126918A1 (en) * | 2020-12-17 | 2022-06-23 | 平安科技（深圳）有限公司 | Method and apparatus for automatically synchronizing associated scripts, and computer device and storage medium |
| CN112612775B (en) * | 2020-12-17 | 2023-06-27 | 平安消费金融有限公司 | Data storage method, device, computer equipment and storage medium |
| CN112612775A (en) * | 2020-12-17 | 2021-04-06 | 平安消费金融有限公司 | Data storage method and device, computer equipment and storage medium |
| CN113448775A (en) * | 2021-06-25 | 2021-09-28 | 中国工商银行股份有限公司 | Multi-source heterogeneous data backup method and device |
| CN114928620A (en) * | 2022-05-31 | 2022-08-19 | 曙光信息产业股份有限公司 | User information synchronization method, device, equipment, storage medium and program product |
| CN114928620B (en) * | 2022-05-31 | 2024-03-12 | 曙光信息产业股份有限公司 | User information synchronization method, apparatus, device, storage medium, and program product |
| CN115033647A (en) * | 2022-08-11 | 2022-09-09 | 杭州湖畔网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN116204509A (en) * | 2023-02-14 | 2023-06-02 | 中电金信软件有限公司 | Data access script generation method and device, electronic equipment and storage medium |
| CN117077218A (en) * | 2023-10-07 | 2023-11-17 | 之江实验室 | Synchronization method, device, medium and equipment for mirror images of multi-center medical institution |
| CN117077218B (en) * | 2023-10-07 | 2024-01-26 | 之江实验室 | Synchronization method, device, medium and equipment for mirror images of multi-center medical institution |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021169277A1 (en) | 2021-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111367883A (en) | Data synchronization method, device, equipment and computer readable storage medium | |
| US11206451B2 (en) | Information interception processing method, terminal, and computer storage medium | |
| CN106874389B (en) | Data migration method and device | |
| EP3382534A1 (en) | Page jumping method and apparatus | |
| WO2020253061A1 (en) | Page generating method and apparatus, computer device, and storage medium | |
| WO2021218143A1 (en) | Data synchronization method and apparatus, and server and storage medium | |
| CA2866449C (en) | Telemetry system for a cloud synchronization system | |
| CN110971655B (en) | Offline client playback and synchronization | |
| CN109522363B (en) | Cloud platform synchronization method, system, equipment and storage medium based on block chain | |
| US9141811B2 (en) | Coerced encryption on connected devices | |
| US20160048703A1 (en) | Securing integrity and consistency of a cloud storage service with efficient client operations | |
| CN110865905A (en) | Data restoration method and device, computer equipment and storage medium | |
| US11048668B2 (en) | Sensitive data management | |
| CN107918564B (en) | Data transmission exception handling method and device, electronic equipment and storage medium | |
| WO2014206089A1 (en) | Terminal mirroring synchronization method, device, terminal and server | |
| CN111339551B (en) | Data verification method and related device and equipment | |
| CN114416883A (en) | Block chain light node data synchronization method, device, equipment and readable storage medium | |
| US20240086632A1 (en) | Methods, system, apparatuses and electronic device for information referencing | |
| CN111327680B (en) | Authentication data synchronization method, device, system, computer equipment and storage medium | |
| JP2015064872A (en) | Monitoring system, system, and monitoring method | |
| US20120173658A1 (en) | Schema Validation Proxy | |
| CN114979109B (en) | Behavior track detection method, behavior track detection device, computer equipment and storage medium | |
| WO2023077748A1 (en) | Account management method and apparatus, and computer device and storage medium | |
| CN114285839B (en) | File transmission method and device, computer storage medium and electronic equipment | |
| CN113609147A (en) | Data sharing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |