CN114254030A - Data synchronization method, system, terminal equipment and storage medium - Google Patents
Data synchronization method, system, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN114254030A CN114254030A CN202111573967.5A CN202111573967A CN114254030A CN 114254030 A CN114254030 A CN 114254030A CN 202111573967 A CN202111573967 A CN 202111573967A CN 114254030 A CN114254030 A CN 114254030A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- task
- database system
- configuration information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application is applicable to the technical field of data processing, and provides a data synchronization method, a data synchronization system and terminal equipment, wherein the data synchronization method comprises the following steps: determining configuration information according to the data synchronization task to configure a synchronization object; establishing and executing a data acquisition task based on the configured source database system and the synchronous data; converting the acquired synchronous data based on a data conversion protocol between synchronous objects; the data distribution task is established based on the converted data and the configured target database system, the data conversion protocol between different database systems is determined through the configuration information, if the interface of the source database system changes or the target database system has new requirements, the data synchronization requirements between different database systems can be realized only by newly adding or changing the configuration information in the configuration file, the interface does not need to be re-developed, the data synchronization efficiency between different database systems is improved, and the resource waste can be reduced.
Description
Technical Field
The present application belongs to the field of data processing technologies, and in particular, to a data synchronization method, system, terminal device, and computer-readable storage medium.
Background
Different database systems have different data structures due to different functions and different development modes, which causes difficulty in data synchronization among different database systems. The traditional cross-system data synchronization mode is usually realized through an interface, that is, an interface is arranged between a target database system and a source database system to realize data calling and synchronization, however, a corresponding interface needs to be developed at one side for calling at the other side, and if the interface of the source database system changes or the target database system has new requirements, the interface needs to be re-developed. Moreover, for the synchronization of the same data source to different systems or nodes, or the synchronization of data collected by different systems or nodes to the same target database system, resources are wasted, and efficiency cannot be guaranteed.
Disclosure of Invention
The embodiment of the application provides a data synchronization method, a data synchronization system, terminal equipment and a computer readable storage medium, which can solve the problems of resource waste and low synchronization efficiency when data synchronization is performed among different types of database systems at present.
In a first aspect, an embodiment of the present application provides a data synchronization method, including:
determining configuration information according to the data synchronization task to configure a synchronization object; the data synchronization task comprises a synchronization object and synchronization data, the synchronization object comprises a source database system and a target database system, and the configuration information comprises a data conversion protocol between the synchronization objects;
establishing and executing a data acquisition task based on the configured source database system and the synchronous data;
converting the acquired synchronous data based on a data conversion protocol between the synchronous objects;
creating a data distribution task based on the converted data and the configured target database system;
and executing the data distribution task to complete data synchronization.
Optionally, the determining configuration information according to the data synchronization task to configure the synchronization object includes:
acquiring a configuration file, and determining a synchronous object according to a data synchronization task;
determining configuration information corresponding to the synchronization object from the configuration file;
and configuring the synchronous object according to the configuration information to obtain a configured source database system and a configured target database system.
Optionally, the data synchronization task is a one-to-many, many-to-one, or many-to-many data synchronization task, and determining configuration information according to the data synchronization task includes:
when the data synchronization task is a one-to-many data synchronization task, determining configuration information of the same source database system and different target database systems;
when the data synchronization task is a many-to-one data synchronization task, determining configuration information of different source database systems and the same target database system;
and when the data synchronization task is a many-to-many data synchronization task, determining configuration information of different source database systems and different target database systems.
Optionally, the creating and executing a data collection task based on the configured source database system and the synchronization data includes:
acquiring the priority of each data acquisition task in a data acquisition task queue;
if the priorities of all the data acquisition tasks in the data acquisition task queue are consistent, determining the execution sequence of the data acquisition tasks based on the creation time of the data acquisition tasks;
and if the priorities of the data acquisition tasks in the acquisition task queue are not consistent, determining the execution sequence of the data acquisition tasks based on the priorities of the data acquisition tasks.
Optionally, the creating a data distribution task based on the converted data and the configured target database system includes:
determining a target field of a target database system according to the converted data;
determining a distribution object according to the configured target database system;
and creating a data distribution task according to the target field of the target database system and the distribution object, and sending the data distribution task to a task distribution queue.
Optionally, before the converting the acquired synchronization data based on the data conversion protocol between the synchronization objects, the method further includes:
judging whether the synchronous database table structure information is consistent;
if the synchronous database table information is consistent, converting the data type according to a data conversion protocol, otherwise, executing the data conversion protocol based on the synchronous objects to convert the acquired synchronous data.
Optionally, after the executing the data distribution task and completing the data synchronization, the method further includes:
and creating a data duplication checking task and sending the data duplication checking task to the target database system.
In a second aspect, an embodiment of the present application provides a data synchronization system, including:
the configuration unit is used for determining configuration information according to the data synchronization task so as to configure the synchronization object; the data synchronization task comprises a synchronization object and synchronization data, the synchronization object comprises a source database system and a target database system, and the configuration information comprises a data conversion protocol between the synchronization objects;
the acquisition unit is used for creating and executing a data acquisition task based on the configured source database system and the synchronous data;
the conversion unit is used for converting the acquired synchronous data based on a data conversion protocol between the synchronous objects;
the distribution unit is used for creating a data distribution task based on the converted data and the configured target database system;
and the synchronization unit is used for executing the data distribution task and completing data synchronization.
In a third aspect, an embodiment of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the data synchronization method according to any one of the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the data synchronization method according to any one of the first aspect.
In a fifth aspect, an embodiment of the present application provides a computer program product, which, when run on a terminal device, enables the terminal device to execute the data synchronization method according to any one of the above first aspects.
Compared with the prior art, the embodiment of the application has the advantages that:
according to the data synchronization method provided by the embodiment of the application, the data conversion protocol and the read-write data interface between different database systems can be determined through the configuration information, if the interface of the source database system changes or the target database system has new requirements, the data synchronization requirements between different database systems can be realized only by newly adding or changing the configuration information in the configuration file, the interface does not need to be re-developed, the data synchronization efficiency between different database systems is improved, the data synchronization requirements of many-to-one, one-to-many and many-to-many can be met, the resource waste is reduced, and the synchronization efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of a data synchronization method according to an embodiment of the present application;
fig. 2 is a schematic flow chart illustrating an implementation of a data synchronization method according to an embodiment of the present application;
fig. 3 is a schematic implementation flow diagram of another data synchronization method provided in an embodiment of the present application;
fig. 4 is a schematic flow chart illustrating an implementation of another data synchronization method provided in an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a data synchronization system provided in an embodiment of the present application;
fig. 6 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to the associated listed items
Any and all possible combinations of one or more of (a) and (b), and including such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
Referring to fig. 1, fig. 1 is a schematic view of an application scenario of a data synchronization method according to an embodiment of the present application. As shown in fig. 1, the main body of the data synchronization method may be the data synchronization system 10 in fig. 1, and the data synchronization system 10 may be communicatively connected to a plurality of database systems 20.
In this embodiment of the application, the data synchronization system 10 may be a terminal device or a server, the terminal device may be an electronic device such as a computer, a mobile phone, a tablet computer, a desktop computer, and a smart wearable device, and the server may be a conventional server or a cloud server, which is not limited specifically herein.
In the embodiment of the present application, the database systems 20 may be different types of database systems, such as SQL SERVER database systems, ORACLE database systems, Mysql database systems, MongoDB database systems, and the like. A user may synchronize data stored across multiple database systems 20 according to actual application needs.
In the embodiment of the present application, the data synchronization method is executed by the data synchronization system 10, so as to collect data to be synchronized from different types of database systems, and synchronize the data into a target database system.
It should be noted that the above application scenarios are only examples for easy understanding, and it should be understood that the embodiments of the present application are not limited to the above application scenarios.
The following will describe in detail the data synchronization method provided in the embodiments of the present application, in which the execution subject of the data synchronization method may be the data synchronization system.
Referring to fig. 2, fig. 2 is a schematic diagram illustrating an implementation flow of a data synchronization method according to an embodiment of the present application. As shown in fig. 2, the data synchronization method provided in an embodiment of the present application may include S11 to S12, which are detailed as follows:
s11: and determining configuration information according to the data synchronization task to configure the synchronization object.
In this embodiment of the present application, the data synchronization task refers to synchronizing synchronization data originally stored in the source database system to the target database system. Specifically, the data synchronization task may be generated according to the data synchronization requirement. For example, the data synchronization task may be generated according to a database system selected by a user and requiring data synchronization, or the corresponding data synchronization task may be generated according to the data synchronization requirement of the business system. For example, the service system may set some data in one or some database systems to be synchronized to other databases at preset time intervals, and at this time, the data synchronization system may generate corresponding data synchronization tasks. It should be noted that the preset time interval may be set according to actual requirements, and is not limited herein.
In this embodiment of the present application, the data synchronization task includes a synchronization object and synchronization data, the synchronization object includes a source database system and a target database system, and the configuration information includes a data conversion protocol between the synchronization objects.
In this embodiment, the configuration information may further include a source database system library instance name, a source database system identification number, a target database system library instance name, a target database system identification number, a data read-write rule, a data conversion protocol, and other contents.
In a specific implementation, configuration information corresponding to a database type may be configured between different database systems, and specifically, a configuration relationship between different systems may be set in a configuration file. For example, assuming that the source database system is an SQL SERVER database system and the target database system is an ORACLE database system, a piece of configuration information about the source database system being the SQL SERVER database system and the target database system being the ORACLE database system may be set in the configuration file, where the configuration information is mainly used to determine a data conversion protocol and a data read-write mode between synchronous objects.
It should be noted that the data conversion protocol is a mapping protocol for converting the data field of the synchronized data in the source database system into the data field in the target database system, and the synchronized reading and writing modes include, but are not limited to, odbc, ole, and io file reading and writing modes.
In one embodiment of the present application, the data synchronization task may be a one-to-many data synchronization task, that is, the data in one source database system is synchronized to a plurality of target database systems, and the determined configuration information is a plurality of configuration information, the method includes that a source database system in a synchronous task corresponds to configuration information of different target database systems, and if the source database system, a first target database system and a second target database system in the synchronous task are SQL SERVER database systems, an ORACLE database system and a Mysql database system, the determined configuration information includes configuration information about the source database system being SQL SERVER database system, configuration information about the ORACLE database system and configuration information about the source database system being SQL SERVER database system, and a target database system being Mysql database system.
In another embodiment of the present application, the data synchronization task may also be a many-to-one synchronization task, that is, data in multiple source database systems are synchronized into one target database system, where the determined configuration information is also multiple configuration information, that is, the configuration information of different source database systems corresponding to the same target database system, and assuming that the source database systems in the synchronization task are SQL SERVER database systems and ORACLE database systems, respectively, and the target database system is a Mysql database system, the determined configuration information includes configuration information about the source database system being a SQL SERVER database system, the target database system being a Mysql database system, and the target database system being a Mysql database system.
In another embodiment of the present application, the data synchronization task may be a many-to-many synchronization task, that is, data in a plurality of source database systems are synchronized into a plurality of target database systems, and the determined configuration information is also a plurality of configuration information, assuming that the source database systems in the synchronization task are SQL SERVER database systems and ORACLE database systems, and the target database systems are Mysql database systems and MongoDB database systems, respectively, the determined configuration information includes SQL SERVER pieces of configuration information about the source database system, the target database system is Mysql database system configuration information, one source database system is SQL SERVER database system, the target database system is MongoDB database system configuration information, one about ORACLE database system is ORACLE database system, the target database system is Mysql database system configuration information, and one about the source database system is acorle database system, the target database system is the configuration information of the MongoDB database system.
In an embodiment of the present application, the step S11 may include the following steps:
acquiring a configuration file, and determining a synchronous object according to a data synchronization task;
determining configuration information corresponding to the synchronization object from the configuration file;
and configuring the synchronous object according to the configuration information to obtain a configured source database system and a configured target database system.
In this embodiment of the present application, the configuration file may be pre-stored in the data synchronization system, and the configuration file includes configuration information between different database systems, where the configuration information may include a source database system library instance name, a source database system identification number, a target database system library instance name, a target database system identification number, a data read-write rule, a data conversion protocol, and other contents.
In the embodiment of the present application, since the synchronization task includes the synchronization object, the source database system and the target database system that need to perform data synchronization can be determined based on the synchronization object field of the synchronization task.
In a specific application, the process of configuring the synchronization object according to the configuration information may include operations of allocating an identification number to the synchronization object, allocating a data read-write interface to the synchronization object, and determining a data conversion protocol between the synchronization objects.
It should be noted that when the data synchronization system obtains the data synchronization task, the data synchronization system may extract a synchronization object in the data synchronization task, that is, the source database system and the target database system, find corresponding configuration information from the configuration file according to the types of the source database system and the target database system, configure the synchronization object according to the configuration information, that is, allocate an identification number to the source database system, allocate an identification number to the target database system, allocate a data reading interface to the source database system, allocate a data writing interface to the target database system, and determine a data conversion protocol between the source database system and the target database system.
For example, assuming that a source database system in a data synchronization task acquired by the data synchronization system is an SQL SERVER database system, and a target database system is a mongoDB database system, the data synchronization system allocates an identification number to the SQL SERVER database system, for example, the allocation identification number is 001, the allocation identification number is 002 for the mongoDB database system, the allocation read data interface to the SQL SERVER database system is an io file read-write interface, the allocation write data interface to the mongoDB database system is an odbc file read-write interface, and it is determined that a data conversion protocol is a data conversion protocol between the SQL SERVER database system and the mongoDB database system.
In an embodiment of the present application, when the data synchronization task is a one-to-many data synchronization task, the data synchronization system respectively allocates identification numbers to a plurality of target database systems for distinguishing different target database systems, and determines a data writing interface of each target database system and a data conversion protocol between the source database system and the different target database systems.
In another embodiment of the present application, when the data synchronization task is a many-to-one data synchronization task, the data synchronization system is also divided into a plurality of source database systems, and identification numbers are respectively allocated to the source database systems for distinguishing different source database systems, and a data reading interface of each source database system and a data conversion protocol between different source database systems and the same target database system are determined.
In another embodiment of the present application, when the data synchronization task is a many-to-many data synchronization task, the data synchronization system respectively allocates identification numbers to the plurality of source database systems to distinguish different source database systems, and respectively allocates identification numbers to the plurality of target database systems to distinguish different target database systems, and can determine a data reading interface of each source database system, a data writing interface of each target database system, and a data conversion protocol between the different source database systems and the different target database systems.
S12: and establishing and executing a data acquisition task based on the configured source database system and the synchronous data.
In this embodiment of the application, the configured source database system determines the identification number, and based on the synchronization data that needs to be synchronized in the data synchronization task, it may determine which data needs to be read from the source database system, and create the data acquisition task based on allocating a read data interface for the source database system, and distribute the identification number of the data acquisition task to the source database system based on the source database system, so as to acquire the synchronization data from the source database system.
In a specific application, the process of executing the data acquisition task includes: and sending the data acquisition task to a source database system corresponding to the identification number through the distributed data reading interface, searching and acquiring corresponding synchronous data by the source database system, and then transmitting the acquired synchronous data to a data synchronization system through the data reading interface, wherein the data synchronization system receives the synchronous data to indicate that the data acquisition task is completed.
In an embodiment of the present application, the data synchronization system may manage the data collection task through the data collection task queue. Specifically, the data collection task queue may sequence the data collection tasks based on the creation time of the data collection tasks, may sequence the data collection tasks based on the priorities of the data collection tasks, and may sequence the data collection tasks based on the data volumes of the data collection tasks, which is not limited herein.
In practical applications, the data collection task queue may also determine the execution order of the data collection tasks based on the priority and creation time. Specifically, when the priorities of the data acquisition tasks in the data acquisition task queue are consistent, the data acquisition tasks are executed based on the sequence of the creation time of the data acquisition tasks, and when the priorities of the data acquisition tasks in the data acquisition task queue are inconsistent, the data acquisition tasks are executed based on the priorities of the data acquisition tasks.
In an embodiment of the present application, the step S12 may include the following steps:
acquiring the priority of each data acquisition task in a data acquisition task queue;
if the priorities of all the data acquisition tasks in the data acquisition task queue are consistent, determining the execution sequence of the data acquisition tasks based on the creation time of the data acquisition tasks;
and if the priorities of the data acquisition tasks in the acquisition task queue are not consistent, determining the execution sequence of the data acquisition tasks based on the priorities of the data acquisition tasks.
In a specific application, the priority of the data acquisition tasks may be determined based on user settings, and a user may set the priority of a data acquisition task synchronized from a certain database system to another database system, and when determining the execution sequence of the data acquisition tasks, the data acquisition task with the higher priority is executed earlier, and the data acquisition task with the lower priority is executed later, that is, the data acquisition task with the highest priority may be set at the head of the data acquisition task queue, and the data acquisition task with the lowest priority may be set at the tail of the data acquisition task queue.
When the priorities of the data acquisition tasks are consistent, determining the execution sequence of the data acquisition tasks based on the creation time of the data acquisition tasks, specifically, the earlier created data acquisition tasks are executed earlier, the later created data acquisition tasks are executed later, that is, the earliest created data acquisition tasks are arranged at the head of the data acquisition task queue, and the latest created data acquisition tasks are arranged at the tail of the data acquisition task queue.
When data acquisition tasks with different priorities exist and a plurality of data acquisition tasks exist in a certain priority, the execution sequence of the data distribution tasks is determined for the plurality of data acquisition tasks in the same priority based on the creation time of the data acquisition tasks, and the data acquisition tasks with different priorities are sorted according to the priorities of the data acquisition tasks, for example, two data acquisition tasks with first priorities, one data acquisition task with second priorities and two data acquisition tasks with third priorities exist. Placing the data acquisition task with early creation time in the data acquisition tasks with the first priority at the head of the data acquisition task queue; setting a data acquisition task with a later time in the data acquisition tasks with the first priority at a second position of the data acquisition task queue; placing the data acquisition tasks of the second priority in a third position of the data acquisition task queue; setting the data collection with early creation time in the data collection tasks with the third priority in the fourth position of the data collection task queue; and setting the data acquisition task with the later time in the data acquisition tasks with the third priority at the tail of the data acquisition task queue.
When there are data collection tasks of different priorities, and there are multiple data collection tasks of a certain priority, the execution order of the data collection tasks is determined for the multiple data collection tasks in the same priority based on the creation time of the data collection tasks.
In an embodiment of the present application, when the data synchronization task is a many-to-one or many-to-many data synchronization task, the data synchronization system may simultaneously create a plurality of data collection tasks.
S13: and converting the acquired synchronous data based on a data conversion protocol.
In this embodiment, after receiving the synchronization data transmitted by the source database system, the data synchronization system can convert the data field and the data format of the synchronization data in the source database system into the data field and the data format of the synchronization data in the target database system according to the data conversion protocol in the configuration information.
In a specific application, the process of converting the acquired synchronous data based on the data conversion protocol may be: and determining a corresponding conversion relation between the data fields in the source database system and the data fields in the target database system based on a data conversion protocol, converting the data fields in the source database system into the data fields in the target database system, and converting the data types in the source database system into the data types in the target database system to complete the data conversion process.
Referring to fig. 3, fig. 3 is a schematic diagram illustrating a flow chart of a data synchronization method according to another embodiment of the present application. In an embodiment of the present application, before S13, the method further includes steps S31 to S32:
s31: judging whether the synchronous database table structure information is consistent;
s32: if the synchronous database table structure information is consistent, converting the data type according to the data conversion protocol; otherwise, S13 is executed.
In the embodiment of the present application, if the database table structure information of the synchronized database is consistent, it indicates that the data fields in the source database system are the same as the data fields in the target database system, so that it is not necessary to convert the data fields in the source database system into the data fields in the target database system, and it is only necessary to convert the data types of the data in the source database system into the data types of the data in the target database system, thereby effectively improving the data synchronization efficiency.
S14: and creating a data distribution task based on the converted data and the configured target database system.
In the embodiment of the application, the configured target database system determines the identification number, and only a data distribution task needs to be created according to the converted data and the data writing interface allocated to the target database system.
It will be appreciated that when the data synchronization task is a one-to-many or many-to-many data synchronization task, the data synchronization system will create multiple data distribution tasks simultaneously.
In an embodiment of the present application, the step S14 may include the following steps:
determining a target field of a target database system according to the converted data;
determining a distribution object according to the configured target database system;
and creating a data distribution task according to the target field of the target database system and the distribution object, and sending the data distribution task to a task distribution queue.
In this embodiment, the distribution object refers to which field of which instance of the target database system the data needs to be inserted.
In a specific implementation, the data distribution task may include a batch data insertion instruction, that is, a data write instruction corresponding to which fields need to be inserted with which data.
In the embodiment of the application, the data distribution task is managed through the task distribution queue. Specifically, the task distribution queue may sequence the data distribution tasks based on the creation time of the data distribution tasks, may sequence the data distribution tasks based on the priorities of the data distribution tasks, and may sequence the data distribution tasks based on the data size of the data distribution tasks, which is not limited herein.
In practical applications, the task distribution queue may determine the execution order of the data distribution tasks based on the priority and creation time. Specifically, when the priorities of the data distribution tasks in the task distribution queue are consistent, the data distribution tasks may be executed based on the time sequence of the creation time for creating the data distribution tasks, and when the priorities of the data distribution tasks in the task distribution queue are inconsistent, the data distribution tasks may be executed based on the priorities of the data distribution tasks.
In an embodiment of the present application, the step S14 may include the following steps:
acquiring the priority of each data distribution task in a task distribution queue;
if the priorities of all the data distribution tasks in the task distribution queue are consistent, determining the execution sequence of the data distribution tasks based on the creation time of the data distribution tasks;
and if the priorities of the data distribution tasks in the task distribution queue are not consistent, determining the execution sequence of the data distribution tasks based on the priorities of the data distribution tasks.
In a specific application, the priority of the data distribution tasks may be determined based on a setting of a user, the user may set a priority of a data collection task synchronized from a certain database system to another database system, and when determining an execution order of the data distribution tasks, the data distribution tasks with higher priorities are executed earlier and the data distribution tasks with lower priorities are executed later, that is, the data distribution task with the highest priority may be set at the head of the task distribution queue, and the data distribution task with the lowest priority may be set at the tail of the task distribution queue.
When the priorities of the data distribution tasks are consistent, the execution order of the data distribution tasks is determined based on the creation time of the data distribution tasks, specifically, the earlier created data distribution task is executed earlier, and the later created data distribution task is executed later, that is, the earliest created data distribution task is arranged at the head of the task distribution queue, and the latest created data distribution task is arranged at the tail of the task distribution queue.
When data distribution tasks with different priorities exist and a plurality of data distribution tasks exist in a certain priority, the execution sequence of the data distribution tasks is determined by the data distribution tasks in the same priority based on the creation time of the data distribution tasks, and the data distribution tasks with different priorities are sorted according to the priorities of the data distribution tasks, for example, two data distribution tasks with first priority, one data distribution task with second priority and two data distribution tasks with third priority exist. Setting a data distribution task with early time in the data distribution tasks with the first priority at the head of the task distribution queue; the data distribution tasks with later time are established in the data distribution tasks with the first priority and are placed at the second position of the task distribution queue; placing the data distribution tasks with the second priority at the third position of the task distribution queue; setting the data distribution task with early time in the data distribution task with the third priority in the fourth position of the task distribution queue; and setting the data distribution task with later time in the data distribution tasks with the third priority at the tail of the task distribution queue.
In an embodiment of the present application, when the same data is synchronized to different target database systems, the target database system may be designated by using the identification number of the target database system, and then different task distribution queues are distinguished based on the identification number of the target database system, that is, a plurality of task distribution queues may be set according to a plurality of target database systems to manage data distribution tasks of corresponding target database systems.
S15: and executing the data distribution task to complete data synchronization.
In the embodiment of the application, the data distribution task is to send the batch data insertion instruction to the target position of the specified target database system, and the target database system executes the data insertion instruction and writes data in the target field to realize data synchronization.
It can be seen from the above that, according to the data synchronization method provided in the embodiment of the present application, the data conversion protocol and the read-write data interface between different database systems are determined by the configuration information, if the interface of the source database system changes or the target database system has a new requirement, the synchronization requirement of data between different database systems can be achieved only by adding or changing the configuration information in the configuration file, the interface does not need to be re-developed, the efficiency of data synchronization between different database systems is improved, meanwhile, the data synchronization requirements of many-to-one, one-to-many, and many-to-many can be met, and the synchronization efficiency is improved while resource waste is reduced.
Referring to fig. 4, fig. 4 is a schematic diagram illustrating an implementation flow of another data synchronization method provided in the embodiment of the present application, and as shown in fig. 4, different from the embodiment shown in fig. 2, the data synchronization method provided in the embodiment of the present application further includes, after S15:
s41: and creating a data duplication checking task and sending the data duplication checking task to the target database system.
In the embodiment of the application, after the data synchronization is realized, in order to avoid data redundancy, the synchronized data can be repeatedly detected in the target database system, and if the synchronized data exists in the target database system, the original data is deleted, and the current synchronized data is reserved.
In a specific implementation, the data synchronization system may send the data duplication checking task to the target database system through a data writing interface configured for the target database system, and the target database system performs operations of data duplication detection and data duplication deletion.
It should be noted that detecting whether data is repeated may be implemented based on an existing manner, which is not described herein again.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Fig. 5 shows a block diagram of a data synchronization system provided in an embodiment of the present application, and for convenience of explanation, only the relevant parts of the embodiment of the present application are shown. Referring to fig. 5, the data synchronization system 50 includes: a configuration unit 51, a collecting unit 52, a converting unit 53, a distributing unit 54 and a synchronizing unit 55. Wherein:
the configuration unit 51 is configured to determine configuration information according to the data synchronization task to configure the synchronization object; the data synchronization task comprises a synchronization object and synchronization data, the synchronization object comprises a source database system and a target database system, and the configuration information comprises a data conversion protocol between the synchronization objects.
The collecting unit 52 is configured to create and execute a data collecting task based on the configured source database system and the synchronization data.
The conversion unit 53 is configured to convert the acquired synchronization data based on a data conversion protocol between the synchronization objects.
The distribution unit 54 is used to create data distribution tasks based on the converted data and the configured target database system.
The synchronization unit 55 is configured to perform the data distribution task and complete data synchronization.
In an embodiment of the present application, the configuration unit 51 includes: the device comprises a file acquisition unit, a determination unit and a processing unit. Wherein:
the file acquisition unit is used for acquiring the configuration file and determining a synchronization object according to the data synchronization task;
the determining unit is used for determining the configuration information corresponding to the synchronous object from the configuration file;
and the processing unit is used for configuring the synchronous object according to the configuration information to obtain a configured source database system and a configured target database system.
In an embodiment of the present application, the data synchronization task is a one-to-many, many-to-one, or many-to-many data synchronization task, and the configuration unit 51 is specifically configured to: when the data synchronization task is a one-to-many data synchronization task, determining configuration information of the same source database system and different target database systems; when the data synchronization task is a many-to-one data synchronization task, determining configuration information of different source database systems and the same target database system; and when the data synchronization task is a many-to-many data synchronization task, determining configuration information of different source database systems and different target database systems.
In an embodiment of the present application, the acquiring unit 52 is specifically configured to: acquiring the priority of each data acquisition task in a data acquisition task queue; if the priorities of all the data acquisition tasks in the data acquisition task queue are consistent, determining the execution sequence of the data acquisition tasks based on the creation time of the data acquisition tasks; and if the priorities of the data acquisition tasks in the acquisition task queue are not consistent, determining the execution sequence of the data acquisition tasks based on the priorities of the data acquisition tasks.
In an embodiment of the application, the distribution unit includes a target field determination unit, a distribution object determination unit, and a task creation unit. Wherein:
the target field determining unit is used for determining a target field of the target database system according to the converted data;
the distribution object determining unit is used for determining a distribution object according to the configured target database system;
and the task creating unit is used for creating a data distribution task according to the target field of the target database system and the distribution object and sending the data distribution task to the task distribution queue.
In an embodiment of the present application, the data synchronization system 50 further includes a determination unit and a data type conversion unit. Wherein:
the judging unit is used for judging whether the synchronous database table structure information is consistent;
and the data type conversion unit is used for converting the data type according to a data conversion protocol if the synchronous database table information is consistent, otherwise, executing the data conversion protocol based on the synchronous objects to convert the acquired synchronous data.
In an embodiment of the present application, the data synchronization system 50 further includes a duplication checking unit.
And the duplication checking unit is used for creating a data duplication checking task and sending the data duplication checking task to the target database system.
It can be seen from the above that, according to the data synchronization system provided in the embodiment of the present application, the data conversion protocol and the read-write data interface between different database systems are determined by the configuration information, if the interface of the source database system changes or the target database system has a new requirement, the synchronization requirement of data between different database systems can be realized only by adding or changing the configuration information in the configuration file, the interface does not need to be re-developed, the efficiency of data synchronization between different database systems is improved, meanwhile, the data synchronization requirements of many-to-one, one-to-many, and many-to-many can be met, and the synchronization efficiency is improved while resource waste is reduced.
Fig. 6 is a schematic structural diagram of a terminal device according to an embodiment of the present application. As shown in fig. 6, the terminal device 6 of this embodiment includes: at least one processor 60 (only one shown in fig. 6), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, the processor 60 implementing the steps in any of the above-described embodiments of the data synchronization method when executing the computer program 62.
Those skilled in the art will appreciate that fig. 6 is only an example of the terminal device 6, and does not constitute a limitation to the terminal device 6, and may include more or less components than those shown, or combine some components, or different components, such as an input/output device, a network access device, and the like.
The Processor 60 may be a Central Processing Unit (CPU), and the Processor 60 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 61 may in some embodiments be an internal storage unit of the terminal device 6, such as a hard disk or a memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are equipped on the terminal device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the terminal device 6. The memory 61 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program. The memory 61 may also be used to temporarily store data that has been output or is to be output.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps in any of the above data synchronization method embodiments may be implemented.
The embodiments of the present application provide a computer program product, which when running on a terminal device, enables the terminal device to implement the steps in any of the above data synchronization method embodiments when executed.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps in any of the above data synchronization method embodiments may be implemented.
The embodiment of the present application provides a computer program product, which, when running on a terminal device, enables the terminal device to implement the steps in any of the above data synchronization method embodiments when executed.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one first processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed data synchronization system and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A method of data synchronization, comprising:
determining configuration information according to the data synchronization task to configure a synchronization object; the data synchronization task comprises a synchronization object and synchronization data, the synchronization object comprises a source database system and a target database system, and the configuration information comprises a data conversion protocol between the synchronization objects;
establishing and executing a data acquisition task based on the configured source database system and the synchronous data;
converting the acquired synchronous data based on a data conversion protocol between the synchronous objects;
creating a data distribution task based on the converted data and the configured target database system;
and executing the data distribution task to complete data synchronization.
2. The data synchronization method of claim 1, wherein the determining configuration information to configure the synchronization object according to the data synchronization task comprises:
acquiring a configuration file, and determining a synchronous object according to a data synchronization task;
determining configuration information corresponding to the synchronization object from the configuration file;
and configuring the synchronous object according to the configuration information to obtain a configured source database system and a configured target database system.
3. The data synchronization method of claim 1, wherein the data synchronization task is a one-to-many, many-to-one, or many-to-many data synchronization task, and the determining the configuration information according to the data synchronization task comprises:
when the data synchronization task is a one-to-many data synchronization task, determining configuration information of the same source database system and different target database systems;
when the data synchronization task is a many-to-one data synchronization task, determining configuration information of different source database systems and the same target database system;
and when the data synchronization task is a many-to-many data synchronization task, determining configuration information of different source database systems and different target database systems.
4. The data synchronization method of claim 1, wherein the creating and executing data collection tasks based on the configured source database system and the synchronization data comprises:
acquiring the priority of each data acquisition task in a data acquisition task queue;
if the priorities of all the data acquisition tasks in the data acquisition task queue are consistent, determining the execution sequence of the data acquisition tasks based on the creation time of the data acquisition tasks;
and if the priorities of the data acquisition tasks in the acquisition task queue are not consistent, determining the execution sequence of the data acquisition tasks based on the priorities of the data acquisition tasks.
5. The data synchronization method of claim 1, wherein creating a data distribution task based on the converted data and the configured target database system comprises:
determining a target field of a target database system according to the converted data;
determining a distribution object according to the configured target database system;
and creating a data distribution task according to the target field of the target database system and the distribution object, and sending the data distribution task to a task distribution queue.
6. The data synchronization method according to any one of claims 1 to 5, further comprising, before the converting the acquired synchronization data based on the data conversion protocol between the synchronization objects:
judging whether the synchronous database table structure information is consistent;
if the synchronous database table information is consistent, converting the data type according to a data conversion protocol, otherwise, executing the data conversion protocol based on the synchronous objects to convert the acquired synchronous data.
7. The data synchronization method of any one of claims 1 to 5, wherein after the performing the data distribution task completes data synchronization, further comprising:
and creating a data duplication checking task and sending the data duplication checking task to the target database system.
8. A data synchronization system, comprising:
the configuration unit is used for determining configuration information according to the data synchronization task so as to configure the synchronization object; the data synchronization task comprises a synchronization object and synchronization data, the synchronization object comprises a source database system and a target database system, and the configuration information comprises a data conversion protocol between the synchronization objects;
the acquisition unit is used for creating and executing a data acquisition task based on the configured source database system and the synchronous data;
the conversion unit is used for converting the acquired synchronous data based on a data conversion protocol between the synchronous objects;
the distribution unit is used for creating a data distribution task based on the converted data and the configured target database system;
and the synchronization unit is used for executing the data distribution task and completing data synchronization.
9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the data synchronization method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the data synchronization method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111573967.5A CN114254030A (en) | 2021-12-21 | 2021-12-21 | Data synchronization method, system, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111573967.5A CN114254030A (en) | 2021-12-21 | 2021-12-21 | Data synchronization method, system, terminal equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114254030A true CN114254030A (en) | 2022-03-29 |
Family
ID=80793826
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111573967.5A Pending CN114254030A (en) | 2021-12-21 | 2021-12-21 | Data synchronization method, system, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114254030A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115277681A (en) * | 2022-07-29 | 2022-11-01 | 山石网科通信技术股份有限公司 | File synchronization method and device for avoiding synchronization process conflict |
-
2021
- 2021-12-21 CN CN202111573967.5A patent/CN114254030A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115277681A (en) * | 2022-07-29 | 2022-11-01 | 山石网科通信技术股份有限公司 | File synchronization method and device for avoiding synchronization process conflict |
| CN115277681B (en) * | 2022-07-29 | 2024-04-02 | 山石网科通信技术股份有限公司 | File synchronization method and device for avoiding conflict in synchronization process |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108959292B (en) | Data uploading method, system and computer readable storage medium | |
| US10007627B2 (en) | Method and apparatus for automatic signal exchange between multiple embedded CPU boards | |
| CN111324610A (en) | Data synchronization method and device | |
| WO2019109518A1 (en) | Data list uploading method and terminal thereof | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| US10282120B2 (en) | Method, apparatus and system for inserting disk | |
| CN112163038A (en) | Cross-cluster data synchronization method, device, equipment and storage medium | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN114254030A (en) | Data synchronization method, system, terminal equipment and storage medium | |
| CN113836184A (en) | Service persistence method and device | |
| CN108234551B (en) | Data processing method and device | |
| CN109005167B (en) | Authentication data processing method and device, server and storage medium | |
| CN110569308A (en) | Data file assembling method, device, equipment and storage medium | |
| CN111209125B (en) | Multi-process command line implementation method | |
| CN110798358B (en) | Distributed service identification method and device, computer readable medium and electronic equipment | |
| CN112506490A (en) | Interface generation method and device, electronic equipment and storage medium | |
| CN113761052A (en) | Database synchronization method and device | |
| CN114791854B (en) | Scheduling method, device, equipment and storage medium of user-state virtual machine task | |
| CN110018987B (en) | Snapshot creating method, device and system | |
| CN108874798B (en) | Big data sorting method and system | |
| CN115665231A (en) | Service creation method, device and computer-readable storage medium | |
| CN116009770A (en) | Read response circuit, method, data transmission system and related equipment | |
| CN115729463A (en) | Data migration method, system and related device | |
| CN114218000A (en) | Log management method and device, electronic equipment and computer readable storage medium | |
| CN111241044B (en) | Method, device, equipment and readable storage medium for constructing heterogeneous database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |