CN111125065A - Visual data synchronization method, system, terminal and computer readable storage medium - Google Patents
Visual data synchronization method, system, terminal and computer readable storage medium Download PDFInfo
- Publication number
- CN111125065A CN111125065A CN201911347664.4A CN201911347664A CN111125065A CN 111125065 A CN111125065 A CN 111125065A CN 201911347664 A CN201911347664 A CN 201911347664A CN 111125065 A CN111125065 A CN 111125065A
- Authority
- CN
- China
- Prior art keywords
- data
- data synchronization
- task
- synchronization
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a visual data synchronization method, a system, a terminal and a computer readable storage medium, wherein the method comprises the following steps: generating a corresponding data synchronization task according to configuration data configured in a visual task configuration interface by a user and sending the data synchronization task to an actuator, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types; and if the data synchronization is not real-time synchronization, the actuator determines and connects the source database according to the source information, and sends the data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task. According to the technical scheme, the interface configuration is friendly, compared with the original operation that a long sqoop script needs to be written in a background, the method is simpler and quicker, and the efficiency of developers can be greatly improved.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a visual data synchronization method, a visual data synchronization system, a visual data synchronization terminal and a computer-readable storage medium.
Background
The most commonly used data processing tool etl (extraction Transformation loading) in the conventional library can extract, convert and load data, but it can only serve the conventional relational library, and cannot meet the direct data import and export requirements of the conventional library and the Hadoop platform. Therefore, in the existing method, data is imported from a traditional library to a Hadoop platform by adopting a sqoop script, although the above problems can be solved, the sqoop script also has some problems, for example, a script command is tedious and is easy to make an error, and the execution of the command is not unified. In addition, the execution results of the past times have no way of statistics, real-time synchronization of data cannot be achieved, and no visual interactive page exists.
Disclosure of Invention
In view of this, embodiments of the present invention provide a visual data synchronization method, system, terminal, and computer-readable storage medium.
An embodiment of the present invention provides a visual data synchronization method, including:
generating a corresponding data synchronization task according to configuration data configured in a visual task configuration interface by a user and sending the data synchronization task to an actuator, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
and if the task synchronization type is non-real-time synchronization, the actuator determines and connects the source database according to the source information, and sends the data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task.
Further, in the above visualized data synchronization method, the method further includes:
if the task synchronization type is real-time synchronization, the data synchronization condition further comprises appointed distributors, and each distributor is associated with a corresponding message processor in advance;
after the executor completes the data synchronization task for the first time, monitoring real-time information of data changes in the source database;
and each time a message with data change is monitored, informing the appointed distributor to distribute the message containing the updating data to the corresponding message processor so that the message processor updates the updating data to the target data table.
Further, in the above visualized data synchronization method, the method further includes:
when the message processor updates the updating data into the target data table, monitoring whether the updating result is in a preset abnormal state;
and if the preset abnormal state is monitored, triggering the strip-by-strip supplementary processing operation of the updated data.
Further, in the above visualized data synchronization method, the data synchronization condition further includes a set number of concurrent tasks and a split field, and if the number of concurrent tasks is greater than 1, the method further includes:
and generating the data synchronization tasks with the quantity of the concurrent tasks according to the split fields and respectively sending the data synchronization tasks to the corresponding quantity of actuators for concurrent execution of the tasks.
Further, in the above visualization data synchronization method, the data synchronization condition further includes a set partition field and a partition policy, and the method further includes:
and dividing a target data table according to the partition field and the partition strategy to obtain a plurality of partitions for storing the data from the source database.
Further, in the visual data synchronization method, the visual task configuration interface includes a source configuration area, a target configuration area, and a data synchronization configuration area; wherein the content of the first and second substances,
the source end configuration area is provided with a source library IP (Internet protocol) to-be-configured item, a source library type to-be-configured item, a source library instance name to-be-configured item, a source library table name to-be-configured item, a source library user name to-be-configured item and a source library password to-be-configured item;
the target end configuration area is provided with a target library type to-be-configured item, a target library name to-be-configured item, a target library table name to-be-configured item, a target library user name to-be-configured item and a target library password to-be-configured item;
and the data synchronization configuration area is provided with a task synchronization type item to be configured and a concurrent task number item to be configured.
Further, in the visual data synchronization method, the source library type included in the source information is a MySQL database or an Oracle database; the target library type contained in the target end information is Hbase database, Kudu database or HDFS database.
Another embodiment of the present invention provides a visual data synchronization system, including:
the visual task configuration module is used for generating a corresponding data synchronization task according to configuration data configured in a visual task configuration interface by a user and sending the data synchronization task to an actuator, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
and the task synchronization execution module is used for determining and connecting the source database according to the source end information if the task synchronization type is non-real-time synchronization, and sending the data in the source database to a target data table determined by the target end information according to the data synchronization condition so as to complete the data synchronization task.
Yet another embodiment of the present invention provides a terminal comprising a processor and a memory, the memory storing a computer program, the processor being configured to execute the computer program to implement the visual data synchronization method described above.
Yet another embodiment of the present invention provides a computer-readable storage medium storing a computer program which, when executed, implements the visual data synchronization method described above.
The visual data synchronization method of the embodiment of the invention realizes the unified specification of the visual interactive interface and the execution command by providing a visual task configuration interface which comprises the setting of the source end information, the target end information and the data synchronization condition, generates the corresponding data synchronization task according to the data configured in the configuration interface by the user, and simultaneously realizes the execution of different data synchronization operations according to different data synchronization conditions, thereby being flexible and easy to modify and the like. Compared with the original operation that long sqoop scripts need to be written in the background, the method is simpler and faster through designing friendly visual configuration interfaces, and the efficiency of developers can be greatly improved.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings required to be used in the embodiments will be briefly described below, and it should be understood that the following drawings only illustrate some embodiments of the present invention, and therefore should not be considered as limiting the scope of the present invention. Like components are numbered similarly in the various figures.
FIG. 1 shows a first flow diagram of a visualization data synchronization method of an embodiment of the invention;
FIG. 2 is a schematic diagram of a visualization task configuration interface of the visualization data synchronization method of an embodiment of the invention;
FIG. 3 shows a second flow diagram of a visualization data synchronization method of an embodiment of the invention;
FIG. 4 is a schematic diagram illustrating an application of the visual data synchronization method according to an embodiment of the present invention;
FIG. 5 is a third flowchart of a visualization data synchronization method according to an embodiment of the present invention;
fig. 6 shows a schematic structural diagram of a visualization data synchronization system according to an embodiment of the present invention.
Description of main component symbols:
100-visual data synchronization system; 110-visualization task configuration module; 120-task synchronous execution module.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
Hereinafter, the terms "including", "having", and their derivatives, which may be used in various embodiments of the present invention, are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be construed as first excluding the existence of, or adding to, one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing.
Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present invention belong. The terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning that is consistent with their contextual meaning in the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein in various embodiments of the present invention.
Example 1
Referring to fig. 1, the embodiment provides a visual data synchronization method, which is applied to migration or real-time synchronization of data in a database, and the like, and by using the method, the direct import and export requirements from a traditional database to a database in a Hadoop platform can be met, and a friendly visual configuration interface is provided, so that the method is simpler and faster, and the efficiency of developers can be greatly improved.
As shown in fig. 1, the visualization data synchronization method is explained in detail below.
And step S110, generating a corresponding data synchronization task according to the configuration data configured in the visual task configuration interface by the user and sending the data synchronization task to an actuator. The configuration data comprises source end information, target end information and data synchronization conditions; the data synchronization condition comprises a set task synchronization type.
Exemplarily, the visualization task configuration interface mainly includes three partial regions, which are a source-side configuration region, a target-side configuration region and a data synchronization configuration region, respectively, as shown in fig. 2. Of course, the three partial areas may also be located in different windows or sub-interfaces, respectively, and a user may switch the windows or sub-interfaces during configuration to implement configuration of source information, a target and data synchronization conditions.
In this embodiment, the source configuration area is mainly provided with a source library IP to-be-configured item, a source library type to-be-configured item, a source library instance name to-be-configured item, a source library table name to-be-configured item, a source library user name to-be-configured item, a source library password to-be-configured item, and the like.
The system determines a source database and a source data table in the source database according to source end information such as a source database IP, a source database type, a source database instance name and the like correspondingly set in the visual task configuration interface by a user, and then establishes connection. Meanwhile, when the system establishes connection with a source database, the system obtains the authority for accessing and data operation of the source database according to the source database user name and the source database password configured by the user.
Further optionally, the source configuration area may include, but is not limited to, further provide: the source table inquires the item to be configured, and/or the source library password file path item to be configured, and the like. The items to be configured of the query conditions of the source table are used for setting the screening conditions of the data to be synchronized, so that the data to be screened is only synchronized. The source library password file path to-be-configured item can be used for setting a password file path, so that the password file path can be displayed, and the password security is improved.
Illustratively, the source library type may include, but is not limited to, MySQL database or Oracle database, etc. It can be understood that, when the user selects different source library types, the source configuration area may further have an item to be configured, which is more detailed than the current source database, according to the source library type selected by the user, which may facilitate system query, and may also facilitate the user to synchronize specific data in a specific data table, and so on.
In this embodiment, the target end configuration area is mainly provided with a target library type to-be-configured item, a target library name to-be-configured item, a target library table name to-be-configured item, a target library user name to-be-configured item, a target library password to-be-configured item, and the like.
The system determines a target database for storing data and a target data table in the target database according to a target library type, a target library name and a target library table name which are correspondingly set in the visual task configuration interface by a user, and establishes connection. Similarly, when the system establishes connection with the target database, the system acquires the access and data operation authority of the target database according to the acquired user name and password information of the target database.
Further optionally, the target configuration area may include, but is not limited to, further including: filter field to be configured, etc. The items to be configured in the filter field can be used for setting sensitive fields in data to realize the elimination of the data of the fields.
In this embodiment, the type of the target library is mainly a database supporting a Hadoop platform. Exemplarily, the target library type may include, but is not limited to, a database supporting a Hadoop platform, such as Hbase database, Kudu database, or HDFS database, and the like.
In this embodiment, the data synchronization configuration area is mainly provided with a task synchronization type item to be configured, a concurrent task number item to be configured, and the like. The system determines whether the task to be generated is a real-time synchronization task or a non-real-time synchronization task according to the task synchronization type set by a user, and determines whether one task or a plurality of concurrently executed tasks are generated according to the set number of concurrent tasks. It can be understood that the plurality of concurrently executed tasks herein refer to dividing the task set this time into several subtasks to be executed simultaneously, and especially when the data size of the data to be synchronized is large, the efficiency of data synchronization can be greatly improved, and the like.
For the non-real-time synchronization task, the system can be executed only once, that is, the data is synchronized only once, and if the synchronization needs to be continued, the execution of the non-real-time synchronization task needs to be triggered manually. Of course, the non-real-time synchronization task may also be timing synchronization, that is, timing execution is performed according to a set time interval, for example, data synchronization is performed every 12 hours, every day, or every week, and the specific synchronization time may be set according to actual requirements. For the real-time synchronization task, after the system executes the first data synchronization, the system also monitors the data update of the source database, and automatically synchronizes the updated data to the target database as long as the data update is monitored.
As another optional situation, for the items to be configured for the number of concurrent tasks in the data synchronization configuration area, if the user sets the number of concurrent tasks greater than 1, the data synchronization configuration area is further provided with the items to be configured for the number of concurrent tasks and the items to be configured for the split field, so that the user can perform task split setting. It can be understood that the processing time of multiple concurrent tasks can be equalized according to the set split field, so that the synchronization efficiency is improved, the pressure of each actuator can be relieved, and the like.
Therefore, if the acquired data synchronization condition further includes a set concurrent task number and a split field, when the concurrent task number is greater than 1, the method further includes: and generating data synchronization tasks with the number equal to the number of the concurrent tasks according to the set splitting field, and then respectively sending the data synchronization tasks to the corresponding number of executors for concurrent execution of the tasks.
On the basis of setting the concurrent task, further optionally, if the data synchronization condition obtained by the system further includes a set partition field and a set partition policy, the method further includes: and dividing the target data table according to the set partition field and the partition strategy so as to obtain a plurality of partitions for storing the data from the source database.
At this time, the data synchronization configuration area is provided with an item to be configured whether to partition or not, so that a user can perform partition storage on the target database or not, and if the user selects the item to be configured whether to partition or not, the data synchronization configuration area is also provided with an item to be configured for partition fields, an item to be configured for partition policies, and the like, so that the user can set corresponding partition fields, partition policies, and the like. For example, the partition field may be set according to a service attribute; the partition policy is used to partition in cooperation with the partition field, for example, the partition field may be intercepted as a partition basis.
It can be understood that the data in the source database can be split and stored in a plurality of different areas according to a plurality of service requirements and the like through partitioning, and if the data volume is large, the subsequent query and analysis efficiency can be improved through reasonable partitioning.
Step S120, if the task synchronization type is non-real-time synchronization, the actuator determines and connects the source database according to the source information, and sends the data in the source database to a target data table determined by the target information according to the data synchronization condition.
For the above step S120, if the synchronization is non-real-time synchronization, the executor executes a history data synchronization, that is, synchronizes the data stored in the source database. Exemplarily, after the source database is connected to the executor according to the obtained source information, data in the source database is migrated to the target data table according to a set data synchronization condition, such as whether concurrent execution is performed, whether partitioned storage is performed, and the like, so as to implement synchronization of the historical data.
As another optional scheme, if the task synchronization type is real-time synchronization, the data synchronization condition further includes a designated distributor, and each distributor associates a corresponding message processor in advance.
As shown in fig. 3, the method further comprises:
step S130, after the executor completes the data synchronization task for the first time, performing real-time message monitoring on the data change in the source database.
Exemplarily, if the real-time synchronization type is set, the system first performs the historical data synchronization operation of step S120, which also provides corresponding data support for subsequent real-time data synchronization, otherwise, after some data in the source database is modified, the target database cannot be updated because the historical data is not migrated. For example, the data update of the source database can be subscribed in a subscription mode, so that message listening is realized.
Step S140, each time a message with data change is monitored, notifying the designated distributor to distribute the message containing the updated data to the corresponding message processor, so that the message processor updates the updated data to the target data table.
Exemplarily, as shown in fig. 4, after the executor executes the first data synchronization task, the data synchronization is continued, and as long as the source database is monitored to have a data update message, the monitored message is pushed to the distributor set by the user and is distributed to the corresponding message processor through the distributor, so as to implement the real-time data synchronization.
Further preferably, when the step S140 is executed, that is, when the message processor updates the update data into the target data table, as shown in fig. 5, the method further includes:
step S150, monitoring whether the update result is in a preset abnormal state, and if the preset abnormal state is monitored, triggering a one-by-one supplementary processing operation on the update data.
The preset abnormal state may include, but is not limited to, a case where the updated value is empty or the writing fails. For the above step S150, if the monitor monitors an abnormal condition, the system will trigger to perform a compensation process on the updated data item by item to update the updated data item again.
The visual data synchronization method provided by this embodiment provides a visual task configuration interface, which includes configuration structure design of source information, target information, and data synchronization conditions, so as to implement unified specification of a visual interaction interface and an execution command, and generate a corresponding data synchronization task according to data configured in the configuration interface by a user; meanwhile, different data synchronization operations are executed according to different data synchronization conditions, whether the data synchronization operations are real-time synchronization operations or not can be set, and the tasks are flexible and easy to modify. Compared with the original method that long sqoop script operations need to be written in the background, the method is simpler and faster through friendly visual interface configuration, and the efficiency of developers can be greatly improved. In addition, the method can realize the data in the traditional database until the data is imported into the database supporting the Hadoop platform, and can solve the problem of functions which are not supported by the existing ETL tool.
Example 2
Referring to fig. 6, based on the visualized data synchronization method of embodiment 1, the present embodiment provides a visualized data synchronization system 100. Wherein the visual data synchronization system 100 comprises:
the visual task configuration module 110 is configured to generate a corresponding data synchronization task according to configuration data configured in a visual task configuration interface by a user, and send the corresponding data synchronization task to an actuator, where the configuration data includes source-end information, target-end information, and data synchronization conditions, and the data synchronization conditions include a set task synchronization type;
and the task synchronization executing module 120 is configured to, if the task synchronization type is non-real-time synchronization, determine and connect the source database according to the source information, and send data in the source database to a target data table determined by the target information according to the data synchronization condition, so as to complete the data synchronization task.
It is understood that the modules in the visualized data synchronization system 100 of the present embodiment correspond to the method of embodiment 1, and the alternatives of embodiment 1 are also applicable to the present embodiment, so that they will not be described in detail herein.
The present invention also provides a terminal, such as a computer, etc., which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program, so that the terminal executes the functions of the above visual data synchronization method or the above modules in the visual data synchronization system 100.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The present invention also provides a computer-readable storage medium for storing the computer program used in the above-mentioned terminal.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention or a part of the technical solution that contributes to the prior art in essence can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention.
Claims (10)
1. A visual data synchronization method is characterized by comprising the following steps:
generating a corresponding data synchronization task according to configuration data configured in a visual task configuration interface by a user and sending the data synchronization task to an actuator, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
and if the task synchronization type is non-real-time synchronization, the actuator determines and connects the source database according to the source information, and sends the data in the source database to a target data table determined by the target information according to the data synchronization condition so as to complete the data synchronization task.
2. The visual data synchronization method according to claim 1, further comprising:
if the task synchronization type is real-time synchronization, the data synchronization condition further comprises appointed distributors, and each distributor is associated with a corresponding message processor in advance;
after the executor completes the data synchronization task for the first time, monitoring real-time information of data changes in the source database;
and each time a message with data change is monitored, informing the appointed distributor to distribute the message containing the updating data to the corresponding message processor so that the message processor updates the updating data to the target data table.
3. The visual data synchronization method according to claim 2, further comprising:
when the message processor updates the updating data into the target data table, monitoring whether the updating result is in a preset abnormal state;
and if the preset abnormal state is monitored, triggering the strip-by-strip supplementary processing operation of the updated data.
4. A visualized data synchronization method according to any one of claims 1 to 3, wherein said data synchronization condition further comprises a set concurrent task number and a split field, and if said concurrent task number is greater than 1, said method further comprises:
and generating the data synchronization tasks with the quantity of the concurrent tasks according to the split fields and respectively sending the data synchronization tasks to the corresponding quantity of actuators for concurrent execution of the tasks.
5. A visual data synchronization method according to any one of claims 1 to 3, wherein the data synchronization condition further comprises a set partition field and a partition policy, the method further comprising:
and dividing a target data table according to the partition field and the partition strategy to obtain a plurality of partitions for storing the data from the source database.
6. The visualization data synchronization method according to claim 1, wherein the visualization task configuration interface comprises a source configuration area, a target configuration area and a data synchronization configuration area; wherein the content of the first and second substances,
the source end configuration area is provided with a source library IP (Internet protocol) to-be-configured item, a source library type to-be-configured item, a source library instance name to-be-configured item, a source library table name to-be-configured item, a source library user name to-be-configured item and a source library password to-be-configured item;
the target end configuration area is provided with a target library type to-be-configured item, a target library name to-be-configured item, a target library table name to-be-configured item, a target library user name to-be-configured item and a target library password to-be-configured item;
and the data synchronization configuration area is provided with a task synchronization type item to be configured and a concurrent task number item to be configured.
7. The synchronization method for visual data according to claim 6, wherein the source library type included in the source information is MySQL database or Oracle database; the target library type contained in the target end information is Hbase database, Kudu database or HDFS database.
8. A visual data synchronization system, comprising:
the visual task configuration module is used for generating a corresponding data synchronization task according to configuration data configured in a visual task configuration interface by a user and sending the data synchronization task to an actuator, wherein the configuration data comprises source end information, target end information and data synchronization conditions, and the data synchronization conditions comprise set task synchronization types;
and the task synchronization execution module is used for determining and connecting the source database according to the source end information if the task synchronization type is non-real-time synchronization, and sending the data in the source database to a target data table determined by the target end information according to the data synchronization condition so as to complete the data synchronization task.
9. A terminal, characterized in that the terminal comprises a processor and a memory, the memory storing a computer program for executing the computer program for implementing the visual data synchronization method of any one of claims 1-7.
10. A computer-readable storage medium, comprising: stored with a computer program which, when executed, implements a visual data synchronization method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911347664.4A CN111125065B (en) | 2019-12-24 | 2019-12-24 | Visual data synchronization method, system, terminal and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911347664.4A CN111125065B (en) | 2019-12-24 | 2019-12-24 | Visual data synchronization method, system, terminal and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111125065A true CN111125065A (en) | 2020-05-08 |
| CN111125065B CN111125065B (en) | 2023-09-12 |
Family
ID=70501921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911347664.4A Active CN111125065B (en) | 2019-12-24 | 2019-12-24 | Visual data synchronization method, system, terminal and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125065B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN113297326A (en) * | 2021-05-21 | 2021-08-24 | 中国邮政储蓄银行股份有限公司 | Data processing method and device, computer readable storage medium and processor |
| CN115017235A (en) * | 2022-06-30 | 2022-09-06 | 上海弘玑信息技术有限公司 | Data synchronization method, electronic device and storage medium |
| CN116339822A (en) * | 2023-05-29 | 2023-06-27 | 安元科技股份有限公司 | Method for simplifying migration of data docking task |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007058517A1 (en) * | 2005-11-21 | 2007-05-24 | Electronics And Telecommunications Research Institute | Method and apparatus for synchronizing visual and voice data in dab/dmb service system |
| US20080005347A1 (en) * | 2006-06-29 | 2008-01-03 | Yahoo! Inc. | Messenger system for publishing podcasts |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN108052681A (en) * | 2018-01-12 | 2018-05-18 | 毛彬 | The synchronous method and system of structural data between a kind of relevant database |
| CN108804613A (en) * | 2018-05-30 | 2018-11-13 | 国网山东省电力公司经济技术研究院 | A kind of Various database real time fusion system and its fusion method |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109359841A (en) * | 2018-09-30 | 2019-02-19 | 福建亿榕信息技术有限公司 | A kind of visual on-line off-line quasi real time synchronous method of archives and system |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN109960710A (en) * | 2019-01-16 | 2019-07-02 | 平安科技（深圳）有限公司 | Method of data synchronization and system between database |
| CN110309161A (en) * | 2019-06-06 | 2019-10-08 | 新华三大数据技术有限公司 | A kind of method of data synchronization, device and server |
-
2019
- 2019-12-24 CN CN201911347664.4A patent/CN111125065B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2007058517A1 (en) * | 2005-11-21 | 2007-05-24 | Electronics And Telecommunications Research Institute | Method and apparatus for synchronizing visual and voice data in dab/dmb service system |
| EP1952629A1 (en) * | 2005-11-21 | 2008-08-06 | Electronics and Telecommunications Research Institute | Method and apparatus for synchronizing visual and voice data in dab/dmb service system |
| US20080005347A1 (en) * | 2006-06-29 | 2008-01-03 | Yahoo! Inc. | Messenger system for publishing podcasts |
| CN106713487A (en) * | 2017-01-16 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data synchronization method and device |
| CN107943979A (en) * | 2017-11-29 | 2018-04-20 | 山东鲁能软件技术有限公司 | The quasi real time synchronous method and device of data between a kind of database |
| CN108052681A (en) * | 2018-01-12 | 2018-05-18 | 毛彬 | The synchronous method and system of structural data between a kind of relevant database |
| CN108804613A (en) * | 2018-05-30 | 2018-11-13 | 国网山东省电力公司经济技术研究院 | A kind of Various database real time fusion system and its fusion method |
| CN109086295A (en) * | 2018-06-13 | 2018-12-25 | 中国平安人寿保险股份有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN109359841A (en) * | 2018-09-30 | 2019-02-19 | 福建亿榕信息技术有限公司 | A kind of visual on-line off-line quasi real time synchronous method of archives and system |
| CN109829009A (en) * | 2018-12-28 | 2019-05-31 | 北京邮电大学 | Configurable isomeric data real-time synchronization and visual system and method |
| CN109960710A (en) * | 2019-01-16 | 2019-07-02 | 平安科技（深圳）有限公司 | Method of data synchronization and system between database |
| CN110309161A (en) * | 2019-06-06 | 2019-10-08 | 新华三大数据技术有限公司 | A kind of method of data synchronization, device and server |
Non-Patent Citations (2)
| Title |
|---|
| QI ZHANG: "Medical Computing Visual Synchronization Using WebSocket Protocol and Web Rendering" * |
| 李成鑫: "备用调度可视化系统的数据同步解决方案初探" * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112287021B (en) * | 2020-07-13 | 2024-04-05 | 上海柯林布瑞信息技术有限公司 | Method and device for generating and synchronizing real-time data synchronization parameters, storage medium and terminal |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN113297326A (en) * | 2021-05-21 | 2021-08-24 | 中国邮政储蓄银行股份有限公司 | Data processing method and device, computer readable storage medium and processor |
| CN115017235A (en) * | 2022-06-30 | 2022-09-06 | 上海弘玑信息技术有限公司 | Data synchronization method, electronic device and storage medium |
| CN116339822A (en) * | 2023-05-29 | 2023-06-27 | 安元科技股份有限公司 | Method for simplifying migration of data docking task |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111125065B (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111125065B (en) | Visual data synchronization method, system, terminal and computer readable storage medium | |
| US11010358B2 (en) | Data migration method and system | |
| US9367598B2 (en) | Merging an out of synchronization indicator and a change recording indicator in response to a failure in consistency group formation | |
| US10585691B2 (en) | Distribution system, computer, and arrangement method for virtual machine | |
| US10685041B2 (en) | Database system, computer program product, and data processing method | |
| US10146633B2 (en) | Data recovery from multiple data backup technologies | |
| US8768973B2 (en) | Apparatus and method for expanding a shared-nothing system | |
| CN106294009A (en) | Database filing method and system | |
| CN108156030B (en) | Method and device for synchronizing configuration strategies | |
| US20160179836A1 (en) | Method for updating data table of keyvalue database and apparatus for updating table data | |
| US20150379038A1 (en) | Data replication in site recovery environment | |
| CN105262835A (en) | Data storage method and device of multiple machine rooms | |
| CN113220431A (en) | Cross-cloud distributed data task scheduling method, device and storage medium | |
| CN112256656A (en) | Transaction rollback method and device, database, system and computer storage medium | |
| CN106372160A (en) | Distributive database and management method | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| CN103744748A (en) | Method for simply, rapidly and automatically backuping FTP (File Transport Protocol) server | |
| CN107168822B (en) | Oracle streams exception recovery system and method | |
| CN105740095B (en) | A kind of method and device of factory reset | |
| CN113254271B (en) | Data sequence recovery method, device, equipment and storage medium | |
| CN109471894B (en) | Novel system and method for uniformly storing distributed files and objects | |
| CN110569231B (en) | Data migration method, device, equipment and medium | |
| CN104778180B (en) | Cloud storage metadata processing system | |
| CN109088921B (en) | Write operation processing method and device and computer readable storage medium | |
| EP3214549A1 (en) | Information processing device, method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |