CN111125065B - Visual data synchronization method, system, terminal and computer readable storage medium - Google Patents
Visual data synchronization method, system, terminal and computer readable storage medium Download PDFInfo
- Publication number
- CN111125065B CN111125065B CN201911347664.4A CN201911347664A CN111125065B CN 111125065 B CN111125065 B CN 111125065B CN 201911347664 A CN201911347664 A CN 201911347664A CN 111125065 B CN111125065 B CN 111125065B
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- data synchronization
- task
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the application discloses a visual data synchronization method, a visual data synchronization system, a visual data synchronization terminal and a computer readable storage medium, wherein the visual data synchronization method comprises the following steps: generating corresponding data synchronization tasks according to configuration data configured by a user in a visual task configuration interface and sending the corresponding data synchronization tasks to an executor, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types; and if the synchronization is non-real-time synchronization, determining and connecting a source database by the executor according to the source information, and transmitting the data in the source database to a target data table determined by target information according to the data synchronization condition so as to complete the data synchronization task. According to the technical scheme, through friendly visual interface configuration, compared with the prior art that lengthy sqoop script operation is required to be written in the background, the method is simpler and faster, and the efficiency of developers and the like can be greatly improved.
Description
Technical Field
The present application relates to the field of big data processing technologies, and in particular, to a visual data synchronization method, a visual data synchronization system, a visual data synchronization terminal, and a computer readable storage medium.
Background
The most commonly used data processing tool ETL (Extraction Transformation Loading) in the conventional library can extract, convert and load data, but can only serve the conventional relational library, and cannot meet the direct data import and export requirements of the conventional library and the Hadoop platform. For this reason, the existing method can solve the above problem by importing data from the conventional library to the Hadoop platform by using the sqoop script, but there are some problems in using the sqoop script, for example, the script command is tedious and error-prone, and the execution of the command is not unified. In addition, the execution results of the past times are not counted, the real-time synchronization of the data cannot be achieved, and visual interactive pages and the like are not available.
Disclosure of Invention
In view of this, embodiments of the present application provide a method, a system, a terminal, and a computer readable storage medium for synchronizing visual data.
An embodiment of the present application provides a method for synchronizing visual data, including:
generating corresponding data synchronization tasks according to configuration data configured by a user in a visual task configuration interface and sending the corresponding data synchronization tasks to an executor, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
and if the task synchronization type is non-real-time synchronization, the executor determines and connects a source database according to the source information, and sends data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task.
Further, in the above visual data synchronization method, the method further includes:
if the task synchronization type is real-time synchronization, the data synchronization condition further comprises designated distributors, and each distributor is pre-associated with a corresponding message processor;
after the executor completes the data synchronization task for the first time, monitoring the data change in the source database by real-time information;
each time a message with a data change is monitored, the designated distributor is notified to distribute a message containing update data to a corresponding message processor, so that the message processor updates the update data into the target data table.
Further, in the above visual data synchronization method, the method further includes:
monitoring whether the updated result is in a preset abnormal state or not when the message processor updates the updated data into the target data table;
and if the preset abnormal state is monitored, triggering the piece-by-piece compensation processing operation of the updated data.
Further, in the above visual data synchronization method, the data synchronization condition further includes a set concurrent task number and a split field, and if the concurrent task number is greater than 1, the method further includes:
and generating the data synchronization tasks with the concurrent task number according to the split field, and respectively sending the data synchronization tasks to the executors with the corresponding number to perform concurrent execution of the tasks.
Further, in the above visual data synchronization method, the data synchronization condition further includes a set partition field and a partition policy, and the method further includes:
and dividing a target data table according to the partition field and the partition strategy to obtain a plurality of partitions for storing data from the source database.
Further, in the above visual data synchronization method, the visual task configuration interface includes a source configuration area, a target configuration area, and a data synchronization configuration area; wherein, the liquid crystal display device comprises a liquid crystal display device,
the source end configuration area is provided with a source library IP to-be-configured item, a source library type to-be-configured item, a source library instance name to-be-configured item, a source library table name to-be-configured item, a source library user name to-be-configured item and a source library password to-be-configured item;
the target end configuration area is provided with a target library type to-be-configured item, a target library name to-be-configured item, a target library table name to-be-configured item, a target library user name to-be-configured item and a target library password to-be-configured item;
and the data synchronization configuration area is provided with a task synchronization type item to be configured and a concurrent task number item to be configured.
Further, in the above visual data synchronization method, the source library type included in the source information is MySQL database or Oracle database; the target library type contained in the target side information is Hbase database, kudu database or HDFS database.
Another embodiment of the present application proposes a visual data synchronization system, including:
the visual task configuration module is used for generating corresponding data synchronization tasks according to configuration data configured in the visual task configuration interface by a user and sending the corresponding data synchronization tasks to the executor, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
and the task synchronization execution module is used for determining and connecting a source database according to the source information if the task synchronization type is non-real-time synchronization, and sending the data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task.
Yet another embodiment of the present application provides a terminal including a processor and a memory storing a computer program for executing the computer program to implement the above-described visual data synchronization method.
Yet another embodiment of the present application provides a computer-readable storage medium storing a computer program which, when executed, implements the above-described visual data synchronization method.
The visual data synchronization method of the embodiment of the application realizes the unified specification of the visual interaction interface and the execution command by providing a visual task configuration interface comprising the setting of source end information, target end information and data synchronization conditions, generates corresponding data synchronization tasks according to the data configured in the configuration interface by a user, and simultaneously realizes the execution of different data synchronization operations according to different data synchronization conditions, thereby being flexible and easy to modify and the like. According to the method, by designing the friendly visual configuration interface, compared with the original method for writing lengthy sqoop script operation in the background, the method is simpler and faster, and the efficiency of developers and the like can be greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the present application, the drawings that are required for the embodiments will be briefly described, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope of the present application. Like elements are numbered alike in the various figures.
FIG. 1 is a schematic diagram of a first process of a visual data synchronization method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a visual task configuration interface of a visual data synchronization method according to an embodiment of the present application;
FIG. 3 is a second flow chart of a visual data synchronization method according to an embodiment of the present application;
FIG. 4 shows an application diagram of a visual data synchronization method according to an embodiment of the present application;
FIG. 5 is a third flow diagram of a visual data synchronization method according to an embodiment of the present application;
fig. 6 shows a schematic structural diagram of a visual data synchronization system according to an embodiment of the present application.
And (3) main component symbol description:
100-a visual data synchronization system; 110-a visual task configuration module; 120-task synchronization execution module.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments.
The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present application.
The terms "comprises," "comprising," "including," or any other variation thereof, are intended to cover a specific feature, number, step, operation, element, component, or combination of the foregoing, which may be used in various embodiments of the present application, and are not intended to first exclude the presence of or increase the likelihood of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing.
Furthermore, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the application belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning that is the same as the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein in connection with the various embodiments of the application.
Example 1
Referring to fig. 1, the present embodiment provides a visual data synchronization method, which is applied to migration or real-time synchronization of data in a database, and the like, by which the direct import and export requirements from a traditional database to a database in a Hadoop platform can be realized, and a friendly visual configuration interface is provided, which is more concise and rapid, and the efficiency of a developer can be greatly improved.
As shown in fig. 1, the visual data synchronization method is described in detail below.
Step S110, corresponding data synchronization tasks are generated according to configuration data configured in the visual task configuration interface by a user and are sent to an executor. The configuration data comprises source terminal information, target terminal information and data synchronization conditions; the data synchronization conditions include a set task synchronization type.
The visual task configuration interface mainly includes three areas, namely a source configuration area, a target configuration area and a data synchronization configuration area, as shown in fig. 2. Of course, the three partial areas may be located in different windows or sub-interfaces, so that the user can switch the windows or sub-interfaces during configuration to realize configuration of source information, target information and data synchronization conditions.
In this embodiment, the source configuration area is mainly provided with a source base IP to-be-configured item, a source base type to-be-configured item, a source base instance name to-be-configured item, a source base table name to-be-configured item, a source base user name to-be-configured item, a source base password to-be-configured item, and the like.
The system determines a source database and a source data table in the source database according to source end information such as source base IP, source base type, source base instance name and the like which are correspondingly set in the visual task configuration interface by a user, and then establishes connection. Meanwhile, when the system establishes connection with the source database, the authority for accessing and operating the data of the source database is obtained according to the source database user name and the source database password configured by the user.
Further alternatively, the source configuration area may include, but is not limited to, being further provided with: the source table queries the condition to be configured items, and/or the source library password file path to be configured items, etc. The source table inquiry condition to-be-configured item is used for setting screening conditions of data to be synchronized so as to realize synchronization of screened data only. The to-be-configured item of the source library password file path can be used for setting a password file path, so that the display in a password file path mode is realized, and the password security and the like are improved.
Illustratively, the source library type may include, but is not limited to, a MySQL database or an Oracle database, or the like. It can be understood that when the user selects different source library types, the source configuration area can be further provided with a to-be-configured item more detailed with the current source database according to the source library type selected by the user, so that the system query can be facilitated, and the user can conveniently synchronize specific data in a specific data table.
In this embodiment, the target configuration area is mainly provided with a target library type to-be-configured item, a target library name to-be-configured item, a target library table name to-be-configured item, a target library user name to-be-configured item, a target library password to-be-configured item, and the like.
The system determines a target database for storing data and a target data table in the target database according to the type, the target library name and the target library table name of the target library correspondingly set in the visual task configuration interface by a user, and establishes connection. Similarly, when a connection is established with the target database, the system acquires the access and data operation authority of the target database according to the acquired user name and password information of the target database.
Further alternatively, the target configuration area may include, but is not limited to, being further provided with: filtering field to be configured items, etc. The filtering field to-be-configured item can be used for setting sensitive fields in data so as to realize the elimination of the data of the fields.
In this embodiment, the target library type is mainly a database supporting a Hadoop platform. Illustratively, the target library type may include, but is not limited to, a database supporting the Hadoop platform, such as an Hbase database, a Kudu database, or an HDFS database.
In this embodiment, the data synchronization configuration area is mainly provided with a task synchronization type to-be-configured item, a concurrent task number to-be-configured item, and the like. The system determines whether the task to be generated is a real-time synchronous task or a non-real-time synchronous task according to the task synchronous type set by a user, and determines whether to generate one task or a plurality of concurrently executed tasks according to the set concurrent task number. It can be understood that, the multiple tasks that are executed concurrently herein refer to dividing the task that is set up at this time into several subtasks to be executed simultaneously, and especially when the data size of the data to be synchronized is large, the efficiency of data synchronization can be greatly improved.
For the above-mentioned non-real-time synchronization task, the system may perform the synchronization of the data only once, and if the synchronization is still needed, the execution of the non-real-time synchronization task needs to be triggered manually. Of course, the non-real-time synchronization task may also be timing synchronization, that is, timing execution is performed at a set time interval, for example, data synchronization is performed every 12 hours, every day or every week, etc., and the specific synchronization time may be set accordingly according to the actual requirement. For the real-time synchronization task, after the system performs the first data synchronization, the system also monitors the data update of the source database, and automatically synchronizes the update data to the target database as soon as the data update is monitored.
As another optional case, for the above-mentioned concurrent task number to-be-configured item in the data synchronization configuration area, if the user sets a concurrent task number greater than 1, the data synchronization configuration area is further provided with a concurrent task number to-be-configured item and a split field to-be-configured item, so that the user can perform task splitting setting. It can be understood that the processing time of a plurality of concurrent tasks can be balanced according to the set split field, so that the synchronization efficiency and the like are improved, and the pressure of each actuator and the like can be relieved.
Then, if the obtained data synchronization condition further includes the set concurrent task number and the split field, when the concurrent task number is greater than 1, the method further includes: and generating a number of data synchronization tasks equal to the number of concurrent tasks according to the set split field, and then respectively sending the data synchronization tasks to a corresponding number of executors to perform concurrent execution of the tasks.
Further optionally, on the basis of setting the concurrent task, if the data synchronization condition acquired by the system further includes a set partition field and a partition policy, the method further includes: and dividing the target data table according to the set partition field and the partition strategy, so as to obtain a plurality of partitions for storing the data from the source database.
In this case, the data synchronization configuration area is provided with a partition to-be-configured item for the user to perform partition storage on the target database, and if the user selects yes in the non-partition to-be-configured item, the data synchronization configuration area is also provided with a partition field to-be-configured item, a partition policy to-be-configured item and the like, so that the user can set corresponding partition fields, partition policies and the like. For example, the partition field may be set according to a service attribute; the partition strategy is used for partitioning in cooperation with the partition field, for example, the partition field can be intercepted as partition basis, etc.
It can be understood that the data in the source database can be split and stored into a plurality of different areas according to a plurality of service demands and the like through partitioning, and if the data volume is large, the reasonable partitioning can also improve the subsequent query and analysis efficiency and the like.
Step S120, if the task synchronization type is non-real-time synchronization, the executor determines and connects to a source database according to the source information, and sends the data in the source database to a target data table determined by the target information according to the data synchronization condition.
For the step S120, if the synchronization is non-real-time, the executor will execute the history data synchronization once, that is, synchronize the data stored in the source database. The executor connects the source information with the source database according to the acquired source information, and then transfers the data in the source database to the target data table according to the set data synchronization conditions, such as whether the data is executed concurrently, whether the data is stored in a partition, and the like, so as to realize the synchronization of the historical data.
As another alternative, if the task synchronization type is real-time synchronization, the data synchronization condition further includes a designated distributor, and each distributor is associated with a corresponding message processor in advance.
As shown in fig. 3, the method further includes:
and step S130, after the executor completes the data synchronization task for the first time, monitoring the data change in the source database by real-time information.
For example, if the real-time synchronization type is set, the system will first execute the history data synchronization operation in step S120, which also provides corresponding data support for the subsequent real-time data synchronization, otherwise, when some data in the source database is modified, the target database cannot be updated because the history data is not migrated. For example, the data update of the source database may be subscribed to by a subscription manner, thereby implementing message listening.
Step S140, each time a message with a data change is monitored, notifying a designated distributor to distribute a message containing update data to a corresponding message processor, so that the message processor updates the update data into the target data table.
As shown in fig. 4, after the executor performs the first data synchronization task, the subsequent data synchronization will continue, and as long as the source database is monitored to have the data updated message, the monitored message is pushed to the distributor set by the user, and is distributed to the corresponding message processor through the distributor, so as to realize the real-time data synchronization.
Further preferably, in performing the step S140, that is, when the message processor updates the update data into the target data table, as shown in fig. 5, the method further includes:
step S150, monitoring whether the updated result is a preset abnormal state, and if so, triggering a piece-by-piece complement processing operation on the updated data.
Illustratively, the preset abnormal state may include, but is not limited to, a case that the updated value is null or the writing fails. For the above step S150, if an abnormal situation is monitored by the monitor, the system triggers to supplement the update data one by one for updating again.
The visual data synchronization method provided by the embodiment provides a visual task configuration interface, wherein the visual task configuration interface comprises configuration structure designs of source end information, target end information and data synchronization conditions, so that a visual interaction interface and unified specification of an execution command are realized, and corresponding data synchronization tasks are generated according to data configured in the configuration interface by a user; meanwhile, different data synchronization operations are implemented according to different data synchronization conditions, whether the data synchronization operations are real-time synchronization operations or not can be set, and tasks are flexibly and easily modified. Compared with the prior art that lengthy sqoop script operation is written in the background, the method is simpler and faster through friendly visual interface configuration, and the efficiency of developers and the like can be greatly improved. In addition, the method can realize the function which is not supported by the existing ETL tool until the data in the traditional database is imported into the database supporting the Hadoop platform.
Example 2
Referring to fig. 6, based on the visual data synchronization method of the above embodiment 1, the present embodiment proposes a visual data synchronization system 100. Wherein the visual data synchronization system 100 comprises:
the visual task configuration module 110 is configured to generate a corresponding data synchronization task according to configuration data configured by a user in the visual task configuration interface, and send the corresponding data synchronization task to the executor, where the configuration data includes source information, target information and data synchronization conditions, and the data synchronization conditions include a set task synchronization type;
and the task synchronization execution module 120 is configured to determine and connect a source database according to the source information if the task synchronization type is non-real-time synchronization, and send data in the source database to a target data table determined by the target information according to the data synchronization condition, so as to complete the data synchronization task.
It will be appreciated that each module in the visual data synchronization system 100 of the present embodiment corresponds to the method of the above embodiment 1, and the options of the above embodiment 1 are also applicable to the present embodiment, so they will not be described in detail herein.
The present application also provides a terminal, such as a computer, etc., which includes a memory storing a computer program and a processor that causes the terminal to execute the above-described visual data synchronization method or the functions of the respective modules in the above-described visual data synchronization system 100 by running the computer program.
The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application program required for a function; the storage data area may store data created according to the use of the terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.
The present application also provides a computer readable storage medium storing the computer program for use in the above terminal.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, of the flow diagrams and block diagrams in the figures, which illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules or units in various embodiments of the application may be integrated together to form a single part, or the modules may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a smart phone, a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application.
Claims (9)
1. A method for synchronizing visual data, comprising:
generating corresponding data synchronization tasks according to configuration data configured by a user in a visual task configuration interface and sending the corresponding data synchronization tasks to an executor, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
if the task synchronization type is non-real-time synchronization, the executor determines and connects a source database according to the source information, and sends data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task;
if the task synchronization type is real-time synchronization, the data synchronization condition further comprises designated distributors, and each distributor is pre-associated with a corresponding message processor;
after the executor completes the data synchronization task for the first time, monitoring the data change in the source database by real-time information;
each time a message with a data change is monitored, the designated distributor is notified to distribute a message containing update data to a corresponding message processor, so that the message processor updates the update data into the target data table.
2. The method for synchronizing visual data according to claim 1, further comprising:
monitoring whether the updated result is in a preset abnormal state or not when the message processor updates the updated data into the target data table;
and if the preset abnormal state is monitored, triggering the piece-by-piece compensation processing operation of the updated data.
3. The visual data synchronization method according to claim 1 or 2, wherein the data synchronization condition further includes a set concurrent task number and a split field, and if the concurrent task number is greater than 1, the method further includes:
and generating the data synchronization tasks with the concurrent task number according to the split field, and respectively sending the data synchronization tasks to the executors with the corresponding number to perform concurrent execution of the tasks.
4. The visualized data synchronization method according to claim 1 or 2, wherein the data synchronization conditions further comprise a set partition field and partition policy, the method further comprising:
and dividing a target data table according to the partition field and the partition strategy to obtain a plurality of partitions for storing data from the source database.
5. The visual data synchronization method according to claim 1, wherein the visual task configuration interface includes a source configuration area, a target configuration area, and a data synchronization configuration area; wherein, the liquid crystal display device comprises a liquid crystal display device,
the source end configuration area is provided with a source library IP to-be-configured item, a source library type to-be-configured item, a source library instance name to-be-configured item, a source library table name to-be-configured item, a source library user name to-be-configured item and a source library password to-be-configured item;
the target end configuration area is provided with a target library type to-be-configured item, a target library name to-be-configured item, a target library table name to-be-configured item, a target library user name to-be-configured item and a target library password to-be-configured item;
and the data synchronization configuration area is provided with a task synchronization type item to be configured and a concurrent task number item to be configured.
6. The visual data synchronization method according to claim 5, wherein the source library type contained in the source information is MySQL database or Oracle database; the target library type contained in the target side information is Hbase database, kudu database or HDFS database.
7. A visual data synchronization system, comprising:
the visual task configuration module is used for generating corresponding data synchronization tasks according to configuration data configured in the visual task configuration interface by a user and sending the corresponding data synchronization tasks to the executor, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
the task synchronization execution module is used for determining and connecting a source database according to the source information if the task synchronization type is non-real-time synchronization, and sending data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task;
the task synchronization execution module is further used for if the task synchronization type is real-time synchronization, the data synchronization condition further comprises designated distributors, and each distributor is pre-associated with a corresponding message processor; after the executor completes the data synchronization task for the first time, monitoring the data change in the source database by real-time information; each time a message with a data change is monitored, the designated distributor is notified to distribute a message containing update data to a corresponding message processor, so that the message processor updates the update data into the target data table.
8. A terminal, characterized in that it comprises a processor and a memory, said memory storing a computer program, said processor being adapted to execute said computer program to implement the visual data synchronization method of any one of claims 1-6.
9. A computer-readable storage medium, comprising: stored with a computer program which, when executed, implements the visual data synchronization method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911347664.4A CN111125065B (en) | 2019-12-24 | 2019-12-24 | Visual data synchronization method, system, terminal and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911347664.4A CN111125065B (en) | 2019-12-24 | 2019-12-24 | Visual data synchronization method, system, terminal and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111125065A CN111125065A (en) | 2020-05-08 |
| CN111125065B true CN111125065B (en) | 2023-09-12 |
Family
ID=70501921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911347664.4A Active CN111125065B (en) | 2019-12-24 | 2019-12-24 | Visual data synchronization method, system, terminal and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125065B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112287021B (en) * | 2020-07-13 | 2024-04-05 | 上海柯林布瑞信息技术有限公司 | Method and device for generating and synchronizing real-time data synchronization parameters, storage medium and terminal |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN113297326A (en) * | 2021-05-21 | 2021-08-24 | 中国邮政储蓄银行股份有限公司 | Data processing method and device, computer readable storage medium and processor |
| CN115017235B (en) * | 2022-06-30 | 2023-07-14 | 上海弘玑信息技术有限公司 | Data synchronization method, electronic device and storage medium |
| CN116339822A (en) * | 2023-05-29 | 2023-06-27 | 安元科技股份有限公司 | Method for simplifying migration of data docking task |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007058517A1 (en) * | 2005-11-21 | 2007-05-24 | Electronics And Telecommunications Research Institute | Method and apparatus for synchronizing visual and voice data in dab/dmb service system |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN108052681A (en) * | 2018-01-12 | 2018-05-18 | 毛彬 | The synchronous method and system of structural data between a kind of relevant database |
| CN108804613A (en) * | 2018-05-30 | 2018-11-13 | 国网山东省电力公司经济技术研究院 | A kind of Various database real time fusion system and its fusion method |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109359841A (en) * | 2018-09-30 | 2019-02-19 | 福建亿榕信息技术有限公司 | A kind of visual on-line off-line quasi real time synchronous method of archives and system |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN109960710A (en) * | 2019-01-16 | 2019-07-02 | 平安科技（深圳）有限公司 | Method of data synchronization and system between database |
| CN110309161A (en) * | 2019-06-06 | 2019-10-08 | 新华三大数据技术有限公司 | A kind of method of data synchronization, device and server |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080005347A1 (en) * | 2006-06-29 | 2008-01-03 | Yahoo! Inc. | Messenger system for publishing podcasts |
-
2019
- 2019-12-24 CN CN201911347664.4A patent/CN111125065B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007058517A1 (en) * | 2005-11-21 | 2007-05-24 | Electronics And Telecommunications Research Institute | Method and apparatus for synchronizing visual and voice data in dab/dmb service system |
| EP1952629A1 (en) * | 2005-11-21 | 2008-08-06 | Electronics and Telecommunications Research Institute | Method and apparatus for synchronizing visual and voice data in dab/dmb service system |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN108052681A (en) * | 2018-01-12 | 2018-05-18 | 毛彬 | The synchronous method and system of structural data between a kind of relevant database |
| CN108804613A (en) * | 2018-05-30 | 2018-11-13 | 国网山东省电力公司经济技术研究院 | A kind of Various database real time fusion system and its fusion method |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109359841A (en) * | 2018-09-30 | 2019-02-19 | 福建亿榕信息技术有限公司 | A kind of visual on-line off-line quasi real time synchronous method of archives and system |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN109960710A (en) * | 2019-01-16 | 2019-07-02 | 平安科技（深圳）有限公司 | Method of data synchronization and system between database |
| CN110309161A (en) * | 2019-06-06 | 2019-10-08 | 新华三大数据技术有限公司 | A kind of method of data synchronization, device and server |
Non-Patent Citations (1)
| Title |
|---|
| 李成鑫.备用调度可视化系统的数据同步解决方案初探.《四川电力技术》.2018,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111125065A (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111125065B (en) | Visual data synchronization method, system, terminal and computer readable storage medium | |
| US11010358B2 (en) | Data migration method and system | |
| CN106325933B (en) | Batch data synchronous method and device | |
| US10685041B2 (en) | Database system, computer program product, and data processing method | |
| US8768973B2 (en) | Apparatus and method for expanding a shared-nothing system | |
| US10467192B2 (en) | Method and apparatus for updating data table in keyvalue database | |
| CN109714188B (en) | Configuration data management method, device and storage medium based on Zookeeper | |
| CN108156030B (en) | Method and device for synchronizing configuration strategies | |
| US9672113B1 (en) | Data recovery from multiple data backup technologies | |
| CN107992491A (en) | A kind of method and device of distributed file system, data access and data storage | |
| US9251189B2 (en) | Update of data structure configured to store metadata associated with a database system | |
| CN106372160A (en) | Distributive database and management method | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| CN108733808B (en) | Big data software system switching method, system, terminal equipment and storage medium | |
| KR101511098B1 (en) | System and method for managing data using distributed containers | |
| US20180293317A1 (en) | Prefix matching using distributed tables for storage services compatibility | |
| CN103905512A (en) | Data processing method and equipment | |
| CN108090152B (en) | Data synchronization method and system | |
| CN109471894B (en) | Novel system and method for uniformly storing distributed files and objects | |
| CN110569231B (en) | Data migration method, device, equipment and medium | |
| CN104778180B (en) | Cloud storage metadata processing system | |
| CN108710665B (en) | Data backflow method, device, system and equipment | |
| JP6197666B2 (en) | Storage device, replication method, and replication program | |
| CN110633332A (en) | Data warehouse, data updating and calling method, device and equipment | |
| US10506038B1 (en) | Method and system for implementing a global node architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |