WO2019000629A1

WO2019000629A1 - Multi-data-source data synchronizing method and system, application server and computer readable storage medium

Info

Publication number: WO2019000629A1
Application number: PCT/CN2017/100051
Authority: WO
Inventors: 付军
Original assignee: 平安科技（深圳）有限公司
Priority date: 2017-06-25
Filing date: 2017-08-31
Publication date: 2019-01-03
Also published as: CN107689982A; CN107689982B

Abstract

A multi-data-source data synchronizing method and system, an application server, and a computer readable storage medium. The method comprising steps of: establishing a first connection between a data platform (4) and at least one data source (5), and establishing a second connection between the data platform (4) and the application server (2) (S41); receiving a source table and types of data sources, selected by a user and needing to be synchronized, and generating table establishing tasks and data synchronizing tasks corresponding to various data sources according to source table structures of the various data sources obtained by pre-resolving (S42); and publishing the generated table establishing tasks and data synchronizing tasks corresponding to the various data sources to a preset workflow scheduling engine, and executing the table establishing tasks and data synchronizing tasks corresponding to the various data sources by means of the preset workflow scheduling engine (S43). The described method can achieve a full-automatic operation of a whole data synchronizing flow, thus reducing the flows of development, testing and deployment for developers.

Description

Multi-data source data synchronization method, system, application server and computer readable storage medium

The present application claims the priority of the Chinese patent application entitled "Multi-data source data synchronization method, application server and computer readable storage medium", which is filed on June 25, 2017, the entire disclosure of which is hereby incorporated by reference. The content is incorporated herein by reference.

Technical field

The present invention relates to the field of computer information technology, and in particular, to a multi-data source data synchronization method, system, application server, and computer readable storage medium.

Background technique

The existing multi-data source data synchronization operation is very complicated, developers need to develop a large number of scripts, and need to go through development, testing, deployment and other processes, long development cycle, large amount of tasks, and error-prone. Therefore, the data synchronization operation of multiple data sources in the prior art is not reasonable enough and needs to be improved.

Summary of the invention

In view of this, the present invention provides a multi-data source data synchronization method, system, application server and computer readable storage medium, which eliminates the development, testing and deployment process of the developer, and the entire data synchronization process realizes fully automatic operation, It saves development costs and reduces the risk of error for developers.

First, in order to achieve the above object, the present invention provides an application server including a memory, a processor, and a multi-data source data synchronization system stored on the memory and operable on the processor, The multi-data source data synchronization system is implemented by the processor to implement the following steps:

Establishing a first connection between the data platform and the at least one data source, and establishing a second connection between the data platform and the application server;

Receiving a source table and a data source type that are selected by the user to be synchronized, and generating a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained by the pre-parsing;

The built-up task and the data synchronization task corresponding to each generated data source are sent to a preset workflow scheduling engine, and the preset workflow scheduling engine executes the table-building task and the data synchronization task corresponding to each data source.

In addition, to achieve the above object, the present invention further provides a multi-data source data synchronization method, which is applied to an application server, and the method includes:

Further, to achieve the above object, the present invention also provides a computer readable storage medium storing a data source data synchronization system, the multiple data source data synchronization system being configurable by at least one processor Execute to implement the following steps:

Further, in order to achieve the above object, the present invention further provides a multi-data source data synchronization system, the system comprising:

a connection module, configured to establish a first connection between the data platform and the at least one data source, and establish a second connection between the data platform and the application server;

The task generation module is configured to receive a source table and a data source type that are selected by the user and need to be synchronized, and generate a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained in advance; and

a data synchronization module, configured to: publish a table-building task and a data synchronization task corresponding to each generated data source to a preset workflow scheduling engine, and execute a table corresponding to each data source by using the preset workflow scheduling engine Task and data synchronization tasks.

Compared with the prior art, the application server, the multi-data source data synchronization method, the system, and the computer-readable storage medium provided by the present invention automatically generate corresponding data sources according to the source table and the data source type selected by the user to be synchronized. And the data synchronization task is automatically published to the preset workflow scheduling engine, and the preset workflow scheduling engine is used to execute the execution center. The task of creating tables and data synchronization tasks realizes one-stop data synchronization service, which eliminates the development, testing and deployment process of developers. The entire data synchronization process realizes fully automatic operation, which saves development costs and reduces developers. The risk of error.

DRAWINGS

1 is a schematic diagram of an optional application environment of each embodiment of the present invention;

2 is a schematic diagram of an optional hardware architecture of the application server of FIG. 1;

3 is a schematic diagram of functional modules of an embodiment of a multi-data source data synchronization system according to the present invention;

FIG. 4 is a schematic diagram of an implementation process of an embodiment of a multi-data source data synchronization method according to the present invention.

Reference mark:

应用服务器 application server	22
网络The internet	33
数据平台Data platform	44
数据源 data source	55
多数据源数据同步系统Multi-data source data synchronization system	2020
存储器Memory	21twenty one
处理器processor	22twenty two
网络接口Network Interface	23twenty three
连接模块 Connection module	201201
任务生成模块 Task generation module	202202
数据同步模块 Data synchronization module	203203
流程步骤Process step	S41-S43S41-S43

The implementation, functional features, and advantages of the present invention will be further described in conjunction with the embodiments.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

It should be noted that the descriptions of "first", "second" and the like in the present invention are for the purpose of description only, and are not to be construed as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. . Thus, features defining "first" and "second" may include at least one of the features, either explicitly or implicitly. In addition, the technical solutions between the various embodiments may be combined with each other, but must be based on the realization of those skilled in the art, and when the combination of the technical solutions is contradictory or impossible to implement, it should be considered that the combination of the technical solutions does not exist. It is also within the scope of protection required by the present invention.

It should be further noted that, in this document, the terms "include", "include" or any of its He is intended to cover non-exclusive inclusions, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or An element inherent in a method, article, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

Referring to FIG. 1 , it is a schematic diagram of an optional application environment according to various embodiments of the present invention.

In the present embodiment, the present invention is applicable to an application environment including, but not limited to, an application server 2, a network 3, a data platform 4, and a data source 5. The application server 2 may be a computing device such as a rack server, a blade server, a tower server, or a rack server. The application server 2 may be an independent server or a server cluster composed of multiple servers. . The network 3 may be an intranet, an Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, Wireless or wired networks such as 5G networks, Bluetooth, Wi-Fi, etc.

The application server 2 is communicably connected to the data platform 4 via the network 3, and the data platform 4 is communicably connected to each data source 5 through the network 3. A first connection (or a first type of connection) is established between the data platform 4 and each data source 5, so that the data platform 4 can perform data transmission with each data source 5 through the first connection. And interact. A second connection (or a second type of connection) is established between the data platform 4 and the application server 2, so that the data platform 4 can perform data transmission and interaction with the application server 2 through the second connection. .

Referring to FIG. 2, it is a schematic diagram of an optional hardware architecture of the application server 2 in FIG. In this embodiment, the application server 2 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can communicate with each other through a system bus. It is noted that FIG. 2 only shows the application server 2 with components 21-23, but it should be understood that not all illustrated components may be implemented, and more or fewer components may be implemented instead.

The memory 21 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), and a random access memory (RAM). , static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. In some embodiments, the memory 21 may be an internal storage unit of the application server 2, such as a hard disk or memory of the application server 2. In other embodiments, the memory 21 may also be an external storage device of the application server 2, such as a plug-in hard disk equipped on the application server 2, a smart memory card (SMC), and a secure digital number. (Secure Digital, SD) card, flash card, etc. Of course, the memory 21 may also include both the internal storage unit of the application server 2 and its external storage device. In this embodiment, the memory 21 is generally used to store an operating system installed in the application server 2 and various types of application software, such as program codes of the multi-data source data synchronization system 20. In addition, The memory 21 can also be used to temporarily store various types of data that have been output or are to be output.

The processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the application server 2, such as performing control and processing related to data interaction or communication with the application server 2. In this embodiment, the processor 22 is configured to run program code or process data stored in the memory 21, such as running the multi-data source data synchronization system 20 and the like.

The network interface 23 may comprise a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the application server 2 and other electronic devices. In this embodiment, the network interface 23 is mainly used to connect the application server 2 to the data platform 4 through the network 3, and establish data transmission between the application server 2 and the data platform 4. Channel and communication connections.

So far, the application environment and the hardware structure and functions of the related devices of the various embodiments of the present invention have been described in detail. Hereinafter, various embodiments of the present invention will be proposed based on the above-described application environment and related equipment.

First, the present invention proposes a multiple data source data synchronization system 20.

Referring to FIG. 3, it is a functional block diagram of an embodiment of the multi-data source data synchronization system 20 of the present invention. In this embodiment, the multi-data source data synchronization system 20 may be divided into one or more modules, the one or more modules being stored in the memory 21 and being processed by one or more processors ( This embodiment is executed by the processor 22) to complete the present invention. For example, in FIG. 3, the multi-data source data synchronization system 20 can be divided into a connection module 201, a task generation module 202, and a data synchronization module 203. The functional module referred to in the present invention refers to a series of computer program instruction segments capable of performing a specific function, and is more suitable than the program to describe the execution process of the multi-data source data synchronization system 20 in the application server 2. The functions of the respective function modules 201-203 will be described in detail below.

The connection module 201 is configured to establish a first connection (or a first type of connection) between the data platform 4 and the at least one data source 5, and establish the data platform 4 and an application server 2 (such as a resolution server). The second connection (or the second type of connection).

Preferably, in the embodiment, the data platform 4 adopts a Hadoop data platform, and establishes a first connection between the data platform 4 and each data source 5 through a data transmission tool in the data platform 4. For example, the data transfer tool is a Sqoop tool set in Hadoop. Those skilled in the art will appreciate that in other embodiments, the data platform and data transfer tool may be of other types.

The task generation module 202 is configured to receive a source table (partial or all) and a data source type that are selected by the user to be synchronized, and generate a table-building task corresponding to each data source according to the source table structure of each data source obtained in advance. Synchronize tasks with data (or as data extraction tasks).

Preferably, in this embodiment, the data source type includes, but is not limited to, a first type of data source (such as an MS SQL Server data source), and a second type of data source (such as an Oracle data source), Three types of data sources (such as MySQL data sources). Those skilled in the art should understand that in other embodiments, the number of data source types may also be increased or decreased according to the requirements of different data platforms. For example, a user may choose to synchronize a portion of the source table in the first type of data source.

Preferably, in this embodiment, the source table structure of the data source 5 includes, but is not limited to, a data type of each data source 5, field information included, and the like.

Preferably, in the embodiment, the application server 2 is provided with a corresponding parsing script file of each data source 5, and the data transfer tool in the data platform 4 is controlled to execute each parsing script file in parallel through a parallel scheduling mechanism. The source table structure parsing task, parsing the source table structure of each data source, and returning the source table structure of each data source obtained by the parsing to the application server 2 through the second connection.

Preferably, in this embodiment, the generating a table-building task corresponding to each data source includes: generating a source table with the selected source table according to a source table (partial or all) and a source table type that are selected by the user to be synchronized. The target table corresponding to the type. The source table type includes, but is not limited to, a first type dimension table (ie, a small amount of data, a source table used by the associated configuration), and a second type task table (ie, the data in the table is continuously updated and added, and the same day Update the historical data except the day), the third type of flow meter (that is, the data in the table is constantly added, and the historical data except the day will not be updated on the same day).

Specifically, if the source table type selected by the user to be synchronized is the first type dimension table (that is, the source table is small, the source table used by the association configuration), the source table structure corresponding to the first type dimension table is generated. a first type of target table, wherein the number of the first type of target tables is determined according to the number of first type source tables selected by the user that need to be synchronized.

If the source table type selected by the user to be synchronized is the second type task table (that is, the data in the table is continuously updated and added, and the historical data except the day is updated on the same day), the second type task table is generated. The second type target table corresponding to the source table structure, wherein the number of the second type target table is determined according to the number of the second type source table that the user selects to synchronize.

If the source table type selected by the user to be synchronized is the third type of flow table (that is, the data in the table is continuously added, and the historical data except the day is not updated on the day), the source of the third type of the flow table is generated. The third type target table corresponding to the table structure, wherein the number of the third type target table is determined according to the number of the third type source table selected by the user to be synchronized.

Preferably, in this embodiment, the generating a data synchronization task corresponding to each data source includes:

Determining the time to extract data from each source table and the time to stop extracting data according to the start time and end time selected by the user that need to be synchronized;

According to the number of source tables that the user needs to synchronize and the target users of the synchronization, the resource configuration (such as CPU, memory, disk, etc.) and resource management nodes required for data synchronization of each task are determined, wherein the resource configuration can be listed by means of a list. Implementation, the resource management node may be implemented by using a Hadoop resource representation model and a resource allocation model;

And establishing a correspondence between the corresponding field of the source table that needs to be synchronized and the corresponding field of each generated target table, and generating a data extraction task corresponding to each source table according to the correspondence between the source table and the corresponding field of the target table.

Specifically, if the source table type selected by the user to be synchronized is the first type dimension table, Corresponding relationship between the corresponding field of the first type dimension table that needs to be synchronized and the corresponding field of each generated first type target table, and according to the corresponding relationship between the first type dimension table and the corresponding field of the first type target table Generating a data extraction task corresponding to each first type dimension table;

If the source table type that is selected by the user to be synchronized is the second type task table, the corresponding relationship between the corresponding field of the second type task table that needs to be synchronized and the corresponding field of each generated second type target table is established. And generating a data extraction task corresponding to each second type task table according to the correspondence between the second type task table and the corresponding field of the second type target table;

If the source table type that is selected by the user to be synchronized is the third type of flow table, the corresponding relationship between the corresponding field of the third type of flow table that needs to be synchronized and the corresponding field of each generated third type target table is established. And generating a data extraction task corresponding to each third type of flow table according to the correspondence between the third type of flow table and the corresponding field of the third type target table.

The data synchronization module 203 is configured to advertise the built-up task and the data synchronization task corresponding to each generated data source to a preset workflow scheduling engine (such as an oozie scheduling engine), and schedule the workflow through the preset workflow. The engine executes the task of creating a table and data synchronization corresponding to each data source.

Preferably, in this embodiment, the performing of the data synchronization task includes:

If the start time that the user needs to synchronize is reached, the data corresponding to each data source is started to synchronize each task, the resource configuration required for each task of data synchronization is acquired, and the data corresponding to the corresponding field is extracted from each source table (metadata information) ), and the extracted data is stored in the corresponding fields of each target table.

Specifically, if the source table type that is selected by the user to be synchronized is the first type dimension table, the first data corresponding to the corresponding field is respectively extracted from each first type dimension table, and the extracted first data is separately stored. Corresponding fields of each type of target table;

If the source table type that is selected by the user to be synchronized is the second type task table, the second data corresponding to the corresponding field is respectively extracted from each second type task table, and the extracted second data is separately stored in each second type. The corresponding field of the target table;

If the source table type selected by the user to be synchronized is the third type of flow table, the third data corresponding to the corresponding field is respectively extracted from each third type of flow table, and the extracted third data is separately stored in each third type. The corresponding field of the target table.

Preferably, in other embodiments, when the multi-data source data synchronization system 20 is executed by the processor 22, the following steps are also implemented:

Controlling, by the parallel scheduling mechanism, the preset workflow scheduling engine to execute the table construction task and the data synchronization task corresponding to each data source in parallel, extracting data corresponding to the corresponding fields from each source table, and storing the extracted data separately Enter the corresponding fields in each target table.

Through the above-mentioned function modules 201-203, the multi-data source data synchronization system 20 of the present invention automatically generates a table-building task and a data synchronization task corresponding to each data source according to the source table and the data source type selected by the user. And automatically generating the table-building task and the data synchronization task corresponding to the generated data source to a preset workflow scheduling engine, and scheduling, by using the preset workflow scheduling engine, the table-building task and the data synchronization task, A one-stop data synchronization service eliminates the development, testing, and deployment process of developers, and the entire data synchronization process enables fully automated operations. It saves development costs and reduces the risk of error for developers.

In addition, the present invention also proposes a multi-data source data synchronization method.

Referring to FIG. 4, it is a schematic flowchart of an implementation manner of an embodiment of a multi-data source data synchronization method according to the present invention. In this embodiment, the order of execution of the steps in the flowchart shown in FIG. 4 may be changed according to different requirements, and some steps may be omitted.

Step S41, establishing a first connection (or a first type of connection) between the data platform 4 and the at least one data source 5, and establishing a second connection between the data platform 4 and an application server 2 (such as a resolution server) ( Or call it the second type of connection).

Step S42, receiving a source table (partial or all) and a data source type that are selected by the user to be synchronized, and generating a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained by pre-parsing (or Call it a data extraction task).

Preferably, in this embodiment, the data source type includes, but is not limited to, a first type of data source (such as an MS SQL Server data source), a second type of data source (such as an Oracle data source), and a third type of data. Source (such as MySQL data source) and so on. Those skilled in the art should understand that in other embodiments, the number of data source types may also be increased or decreased according to the requirements of different data platforms. For example, a user may choose to synchronize a portion of the source table in the first type of data source.

Preferably, in this embodiment, the generating a table-building task corresponding to each data source includes: generating a source table with the selected source table according to a source table (partial or all) and a source table type that are selected by the user to be synchronized. The target table corresponding to the type. Preferably, in this embodiment, the source table type includes, but is not limited to, a first type dimension table (ie, a small amount of data, a source table used by an associated configuration), and a second type task table (ie, data in the table continuously) There are updates and additions, and the historical data except the day will be updated on the same day), and the third type of flow table (that is, the data in the table is continuously added, and the historical data except the day is not updated on the same day).

Specifically, if the source table type selected by the user to be synchronized is the first type dimension table (that is, the source table is small, the source table used by the association configuration), the source table structure corresponding to the first type dimension table is generated. a first type of target table, wherein the number of the first type of target tables is selected according to the user The number of source tables that need to be synchronized is determined by the number of source tables.

Specifically, if the source table type that the user selects to synchronize is the first type dimension table, the corresponding field of the first type dimension table that needs to be synchronized and the corresponding field of each generated first type target table are established. Corresponding relationship, and generating a data extraction task corresponding to each first type dimension table according to the correspondence between the first type dimension table and the corresponding field of the first type target table;

Step S43: Publish the built-up task and the data synchronization task corresponding to the generated data sources to a preset workflow scheduling engine (such as an oozie scheduling engine), and execute corresponding data sources by using the preset workflow scheduling engine. The task of building tables and data synchronization tasks.

Preferably, in other embodiments, the method further comprises the steps of:

Through the above steps S41-S43, the multi-data source data synchronization method proposed by the present invention automatically generates a table-building task and a data synchronization task corresponding to each data source according to the source table and the data source type selected by the user to be synchronized, and The generated table-building task and the data synchronization task corresponding to each generated data source are automatically advertised to a preset workflow scheduling engine, and the preset workflow task scheduling engine executes the table-building task and the data synchronization task to implement one The station-based data synchronization service eliminates the developer's development, testing, and deployment processes, and the entire data synchronization process enables fully automated operations, saving development costs and reducing developer error risk.

Further, in order to achieve the above object, the present invention further provides a computer readable storage medium (such as a ROM/RAM, a magnetic disk, an optical disk), the computer readable storage medium storing a multiple data source data synchronization system 20, The multiple data source data synchronization system 20 can be executed by at least one processor 22 to cause the at least one processor to perform the steps of the multiple data source data synchronization method as described above.

Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and can also be implemented by hardware, but in many cases, the former is A better implementation. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, The optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present invention.

The preferred embodiments of the present invention have been described above with reference to the drawings, and are not intended to limit the scope of the invention. The serial numbers of the embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments. Additionally, although logical sequences are shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than the ones described herein.

A person skilled in the art can implement the invention in various variants without departing from the scope and spirit of the invention. For example, the features of one embodiment can be used in another embodiment to obtain a further embodiment. Equivalent structure or equivalent process transformation using the description of the invention and the contents of the drawings, or directly Or indirectly, in other related technical fields, it is equally included in the scope of patent protection of the present invention.

Claims

An application server, comprising: a memory, a processor, and a multi-data source data synchronization system stored on the memory and operable on the processor, the multi-data source data synchronization system The following steps are implemented when executed by the processor:

Establishing a first connection between the data platform and the at least one data source, and establishing a second connection between the data platform and the application server;

Receiving a source table and a data source type that are selected by the user to be synchronized, and generating a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained by the pre-parsing;

The built-up task and the data synchronization task corresponding to each generated data source are sent to a preset workflow scheduling engine, and the preset workflow scheduling engine executes the table-building task and the data synchronization task corresponding to each data source.
The application server according to claim 1, wherein the generating a table-building task corresponding to each data source comprises: generating a source table with the selected source table according to a source table and a source table type that are selected by the user to be synchronized The target table corresponding to the type.
The application server according to claim 2, wherein the generating a table-building task corresponding to each data source comprises:

If the source table type that the user selects to synchronize is the first type dimension table, generate a first type target table corresponding to the source table structure of the first type dimension table, and the number of the first type target table is determined according to the Determining the number of source types of the first type that the user selects to synchronize;

If the source table type that the user selects to synchronize is the second type task table, generate a second type target table corresponding to the source table structure of the second type task table, and the number of the second type target table is determined according to the Determining the number of source types selected by the user that need to be synchronized; and

If the source table type that the user selects to synchronize is the third type of flow table, generate a third type target table corresponding to the source table structure of the third type of flow table, and the number of the third type of target table is determined according to the The number of source tables selected by the user that need to be synchronized is determined.
The application server according to claim 2 or 3, wherein the generating a data synchronization task corresponding to each data source comprises:

Determining the time to extract data from each source table and the time to stop extracting data according to the start time and end time selected by the user that need to be synchronized;

According to the number of source tables selected by the user to be synchronized and the synchronized target end users, the resource configuration and resource management nodes required for data synchronization tasks are determined;

And establishing a correspondence between the corresponding field of the source table that needs to be synchronized and the corresponding field of each generated target table, and generating a data extraction task corresponding to each source table according to the correspondence between the source table and the corresponding field of the target table.
The application server method according to claim 4, wherein the execution of the data synchronization task comprises:

If the start time that the user needs to synchronize is reached, the data corresponding to each data source is started to synchronize each task, the resource configuration required for each task of data synchronization is obtained, and the data corresponding to the corresponding field is extracted from each source table, and the data is extracted. The data is stored in the corresponding fields of each target table.
A multi-data source data synchronization method is applied to an application server, and the method includes:

Establishing a first connection between the data platform and the at least one data source, and establishing a second connection between the data platform and the application server;

Receiving a source table and a data source type that are selected by the user to be synchronized, and generating a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained by the pre-parsing;

The built-up task and the data synchronization task corresponding to each generated data source are sent to a preset workflow scheduling engine, and the preset workflow scheduling engine executes the table-building task and the data synchronization task corresponding to each data source.
The multi-data source data synchronization method according to claim 6, wherein the generating a table-building task corresponding to each data source comprises: generating, according to a source table and a source table type that are selected by the user to be synchronized The target table corresponding to the selected source table type.
The multi-data source data synchronization method according to claim 7, wherein the generating a table-building task corresponding to each data source comprises:

If the source table type that the user selects to synchronize is the first type dimension table, generate a first type target table corresponding to the source table structure of the first type dimension table, and the number of the first type target table is determined according to the Determining the number of source types of the first type that the user selects to synchronize;

If the source table type that the user selects to synchronize is the second type task table, generate a second type target table corresponding to the source table structure of the second type task table, and the number of the second type target table is determined according to the Determining the number of source types selected by the user that need to be synchronized; and

If the source table type that the user selects to synchronize is the third type of flow table, generate a third type target table corresponding to the source table structure of the third type of flow table, and the number of the third type of target table is determined according to the The number of source tables selected by the user that need to be synchronized is determined.
The multi-data source data synchronization method according to claim 7 or 8, wherein the generating a data synchronization task corresponding to each data source comprises:

Determining the time to extract data from each source table and the time to stop extracting data according to the start time and end time selected by the user that need to be synchronized;

According to the number of source tables selected by the user to be synchronized and the synchronized target end users, the resource configuration and resource management nodes required for data synchronization tasks are determined;

Establishing a corresponding field of the source table that needs to be synchronized with a corresponding field of each generated target table The corresponding relationship, and the data extraction task corresponding to each source table is generated according to the correspondence between the source table and the corresponding field of the target table.
The multi-data source data synchronization method according to claim 9, wherein the performing of the data synchronization task comprises:

If the start time that the user needs to synchronize is reached, the data corresponding to each data source is started to synchronize each task, the resource configuration required for each task of data synchronization is obtained, and the data corresponding to the corresponding field is extracted from each source table, and the data is extracted. The data is stored in the corresponding fields of each target table.
A computer readable storage medium, characterized in that the computer readable storage medium stores a multi data source data synchronization system, the multi data source data synchronization system being executable by at least one processor to implement the following steps:

Establishing a first connection between the data platform and the at least one data source, and establishing a second connection between the data platform and the application server;

Receiving a source table and a data source type that are selected by the user to be synchronized, and generating a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained by the pre-parsing;

The built-up task and the data synchronization task corresponding to each generated data source are sent to a preset workflow scheduling engine, and the preset workflow scheduling engine executes the table-building task and the data synchronization task corresponding to each data source.
The computer readable storage medium according to claim 11, wherein the generating a table-building task corresponding to each data source comprises: generating and selecting according to a source table and a source table type that are selected by the user to be synchronized. The target table corresponding to the source table type.
The computer readable storage medium according to claim 12, wherein the generating a table-building task corresponding to each data source comprises:

If the source table type that the user selects to synchronize is the first type dimension table, generate a first type target table corresponding to the source table structure of the first type dimension table, and the number of the first type target table is determined according to the Determining the number of source types of the first type that the user selects to synchronize;

If the source table type that the user selects to synchronize is the second type task table, generate a second type target table corresponding to the source table structure of the second type task table, and the number of the second type target table is determined according to the Determining the number of source types selected by the user that need to be synchronized; and

If the source table type that the user selects to synchronize is the third type of flow table, generate a third type target table corresponding to the source table structure of the third type of flow table, and the number of the third type of target table is determined according to the The number of source tables selected by the user that need to be synchronized is determined.
The computer readable storage medium according to claim 12 or 13, wherein the generating a data synchronization task corresponding to each data source comprises:

According to the start time and end time that the user selects to synchronize, it is determined to start from each source table. The time of the data and the time at which the data is stopped;

According to the number of source tables selected by the user to be synchronized and the synchronized target end users, the resource configuration and resource management nodes required for data synchronization tasks are determined;

And establishing a correspondence between the corresponding field of the source table that needs to be synchronized and the corresponding field of each generated target table, and generating a data extraction task corresponding to each source table according to the correspondence between the source table and the corresponding field of the target table.
The computer readable storage medium of claim 14, wherein the performing of the data synchronization task comprises:

If the start time that the user needs to synchronize is reached, the data corresponding to each data source is started to synchronize each task, the resource configuration required for each task of data synchronization is obtained, and the data corresponding to the corresponding field is extracted from each source table, and the data is extracted. The data is stored in the corresponding fields of each target table.
A multi-data source data synchronization system, characterized in that the system comprises:

a connection module, configured to establish a first connection between the data platform and the at least one data source, and establish a second connection between the data platform and the application server;

The task generation module is configured to receive a source table and a data source type that are selected by the user and need to be synchronized, and generate a table construction task and a data synchronization task corresponding to each data source according to the source table structure of each data source obtained in advance; and

a data synchronization module, configured to: publish a table-building task and a data synchronization task corresponding to each generated data source to a preset workflow scheduling engine, and execute a table corresponding to each data source by using the preset workflow scheduling engine Task and data synchronization tasks.
The multi-data source data synchronization system according to claim 16, wherein the generating a table-building task corresponding to each data source comprises: generating, according to a source table and a source table type that are selected by the user to be synchronized. The target table corresponding to the selected source table type.
The multi-data source data synchronization system according to claim 17, wherein the generating a table-building task corresponding to each data source comprises:

If the source table type that the user selects to synchronize is the first type dimension table, generate a first type target table corresponding to the source table structure of the first type dimension table, and the number of the first type target table is determined according to the Determining the number of source types of the first type that the user selects to synchronize;

If the source table type that the user selects to synchronize is the second type task table, generate a second type target table corresponding to the source table structure of the second type task table, and the number of the second type target table is determined according to the Determining the number of source types selected by the user that need to be synchronized; and

If the source table type that the user selects to synchronize is the third type of flow table, generate a third type target table corresponding to the source table structure of the third type of flow table, and the number of the third type of target table is determined according to the The number of source tables selected by the user that need to be synchronized is determined.
The multi-data source data synchronization system according to claim 17 or 18, wherein the generating a data synchronization task corresponding to each data source comprises:

Determining the time to extract data from each source table and the time to stop extracting data according to the start time and end time selected by the user that need to be synchronized;

According to the number of source tables selected by the user to be synchronized and the synchronized target end users, the resource configuration and resource management nodes required for data synchronization tasks are determined;

And establishing a correspondence between the corresponding field of the source table that needs to be synchronized and the corresponding field of each generated target table, and generating a data extraction task corresponding to each source table according to the correspondence between the source table and the corresponding field of the target table.
The multi-data source data synchronization system of claim 19, wherein the performing of the data synchronization task comprises:

If the start time that the user needs to synchronize is reached, the data corresponding to each data source is started to synchronize each task, the resource configuration required for each task of data synchronization is obtained, and the data corresponding to the corresponding field is extracted from each source table, and the data is extracted. The data is stored in the corresponding fields of each target table.