CN112463762A - Method, system, device and medium for cross-cluster real-time data migration and synchronization - Google Patents
Method, system, device and medium for cross-cluster real-time data migration and synchronization Download PDFInfo
- Publication number
- CN112463762A CN112463762A CN202011231345.XA CN202011231345A CN112463762A CN 112463762 A CN112463762 A CN 112463762A CN 202011231345 A CN202011231345 A CN 202011231345A CN 112463762 A CN112463762 A CN 112463762A
- Authority
- CN
- China
- Prior art keywords
- cluster
- source
- data
- middleware
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 110
- 230000005012 migration Effects 0.000 title claims abstract description 26
- 238000013508 migration Methods 0.000 title claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 71
- 230000005540 biological transmission Effects 0.000 claims abstract description 24
- 230000004044 response Effects 0.000 claims abstract description 24
- 230000008676 import Effects 0.000 claims description 19
- 230000005856 abnormality Effects 0.000 claims description 15
- 230000000977 initiatory effect Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 241000238876 Acari Species 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
Abstract
The invention discloses a method, a system, equipment and a storage medium for transferring and synchronizing cross-cluster real-time data, wherein the method comprises the following steps: in response to receiving the synchronization task, analyzing the synchronization task and generating a configuration file, and issuing the configuration file to directories corresponding to the source cluster and the target cluster to determine a transmission path; starting a first process of a corresponding type in the source cluster to lead data acquired from a source component of the source cluster into a source cluster middleware; starting a second process among the clusters to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and starting a third process of a corresponding type in the target cluster to lead the data in the middleware of the target cluster to different target components in real time. The invention starts different processes by using the middleware, thereby being capable of carrying out data migration and synchronization among different clusters, and having simple operation and strong universality.
Description
Technical Field
The present invention relates to the field of data synchronization, and more particularly, to a method, a system, a computer device, and a readable medium for cross-cluster real-time data migration and synchronization.
Background
Currently, many large internet services are very complex, often need to be supported by multiple clusters or even multiple data centers, and in a multi-cluster scenario, operations such as data backup, synchronization, migration and the like are usually involved. Big difference is huge between the big data assembly, can't be completely compatible each other, and it also needs multiple complicated loaded down with trivial details operation to synchronize data before different big data assembly, for example: the synchronization of HDFS (Hadoop Distributed File System) data to MySql (database) requires the additional introduction of sqoop (a data transmission tool), and the editing of complex commands, while once a component is replaced, a new path is required, and a general pipeline is lacking for data synchronization and migration. Furthermore, synchronization by mobile storage devices is not, but not efficient and less secure, and does not support real-time synchronization.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, a system, a computer device, and a computer readable storage medium for performing cross-cluster real-time data migration and synchronization, which combine different processes with middleware in a cluster to achieve the purpose of synchronizing data in different clusters, have a simple structure, are highly easy to use and convenient to maintain, provide a relatively universal cluster migration scheme for different big data components, and provide great convenience for internet big data services.
Based on the above object, an aspect of the embodiments of the present invention provides a method for migrating and synchronizing data in real time across clusters, including the following steps: in response to receiving a synchronization task, analyzing the synchronization task and generating a configuration file, and issuing the configuration file to directories corresponding to a source cluster and a target cluster to determine a transmission path; starting a first process of a corresponding type in the source cluster to lead data acquired from a source component of the source cluster into a source cluster middleware; starting a second process among the clusters to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and starting a third process of a corresponding type in the target cluster to lead data in the middleware of the target cluster to different target components in real time.
In some embodiments, the initiating, in the source cluster, a first process of a corresponding type to import data obtained from a source component of the source cluster into source cluster middleware comprises: judging whether the source component and the target component are of the same type; and in response to the source component not being of the same type as the target component, converting data of the source component into a format that is compliant with the target component.
In some embodiments, the initiating, in the source cluster, a first process of a corresponding type to import data obtained from a source component of the source cluster into source cluster middleware comprises: and adding a time stamp of the moment of importing the source cluster middleware into the data.
In some embodiments, the starting, in the target cluster, a third process of a corresponding type to import data in the target cluster middleware to a different target component in real time includes: acquiring a timestamp of the data, and judging whether a difference value between a time corresponding to the timestamp and the current time exceeds a threshold value; and responding to the difference value between the time corresponding to the timestamp and the current time exceeding a threshold value, searching for abnormality and giving an alarm.
In some embodiments, the initiating a second process between clusters to synchronize data in the source cluster middleware to the target cluster middleware in real time comprises: and adding a second time stamp of the time of importing the middleware of the target cluster into the data.
In some embodiments, the finding an anomaly and alerting comprises: judging whether the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value; and determining that the abnormality is the second process and giving an alarm in response to the fact that the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value.
In some embodiments, the finding an anomaly and alerting comprises: in response to that the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp does not exceed a second threshold, judging whether the difference between the current time and the time corresponding to the second timestamp exceeds a third threshold; and determining that the abnormality is the third process and giving an alarm in response to the fact that the difference value between the current time and the time corresponding to the second timestamp exceeds a third threshold value.
In another aspect of the embodiments of the present invention, a cross-cluster real-time data migration and synchronization system is further provided, including: the analysis module is configured to respond to the received synchronization task, analyze the synchronization task, generate a configuration file, and issue the configuration file to directories corresponding to the source cluster and the target cluster to determine a transmission path; the first transmission module is configured to start a first process of a corresponding type in the source cluster so as to lead data acquired from a source component of the source cluster into a source cluster middleware; the second transmission module is configured to start a second process among the clusters so as to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and a third transmission module, configured to start a third process of a corresponding type in the target cluster to import data in the middleware of the target cluster to different target components in real time.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method as above.
In a further aspect of the embodiments of the present invention, a computer-readable storage medium is also provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has the following beneficial technical effects: the purpose of synchronizing data in different clusters is achieved by combining middleware in the clusters with different processes, the structure is simple, the usability is strong, the maintenance is convenient, a relatively universal cluster migration scheme is provided for different big data assemblies, and great convenience is provided for Internet big data services.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic diagram of an embodiment of a cross-cluster real-time data migration and synchronization method provided by the present invention;
fig. 2 is a schematic hardware structure diagram of an embodiment of a cross-cluster real-time data migration and synchronization computer device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In view of the foregoing, a first aspect of the embodiments of the present invention provides an embodiment of a method for migrating and synchronizing cross-cluster real-time data. Fig. 1 is a schematic diagram illustrating an embodiment of a cross-cluster real-time data migration and synchronization method provided by the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, responding to the received synchronization task, analyzing the synchronization task and generating a configuration file, and issuing the configuration file to the directories corresponding to the source cluster and the target cluster to determine a transmission path;
s2, starting a first process of a corresponding type in the source cluster to lead the data acquired from the source component of the source cluster into the middleware of the source cluster;
s3, starting a second process among the clusters to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and
and S4, starting a third process of a corresponding type in the target cluster to lead the data in the middleware of the target cluster to different target components in real time.
And responding to the received synchronization task, analyzing the synchronization task, generating a configuration file, and issuing the configuration file to the directories corresponding to the source cluster and the target cluster to determine a transmission path. When receiving the synchronous task, the task parameters can be analyzed, and the configuration file is generated and remotely issued to the specific directories of the source cluster and the target cluster.
A first process of a corresponding type is started in the source cluster to import data obtained from a source component of the source cluster into the source cluster middleware. The middleware may be, for example, Kafka, and the first process of a different type, for example, a Source Connector process, is started under the Kafka service of the Source cluster, and is used as a data pipeline to import data from each component to Kafka. For example, starting HDFS sourceConnector may import data under HDFS specific directory into Kafka Topic (subject)
A second process between the clusters is started to synchronize data in the source cluster middleware to the target cluster middleware in real time. The second process may be, for example, a Kafka MirrorMaker (mirror synchronization) process that starts the Kafka MirrorMaker as an inter-cluster data pipe to synchronize Kafka data of the source cluster to Kafka Topic of the target cluster in real time. And data migration synchronization of a single cluster to multiple clusters is supported, namely the source cluster is single and the target cluster is multiple.
And starting a third process of a corresponding type in the target cluster to lead the data in the middleware of the target cluster to different target components in real time. The third process may be, for example, a Sink Connector process, where different types of Sink connectors are started in the target cluster, and the Sink connectors are used as data pipelines to import data from Kafka to different data components in real time.
In some embodiments, the initiating, in the source cluster, a first process of a corresponding type to import data obtained from a source component of the source cluster into source cluster middleware comprises: judging whether the source component and the target component are of the same type; and in response to the source component not being of the same type as the target component, converting data of the source component into a format that is compliant with the target component. Based on the conversion capability of Kafka Connector, Source and Sink may correspond to different big data components, for example, HDFS data of cluster 1 may be migrated and synchronized to MySql of cluster 2.
In some embodiments, the initiating, in the source cluster, a first process of a corresponding type to import data obtained from a source component of the source cluster into source cluster middleware comprises: and adding a time stamp of the moment of importing the source cluster middleware into the data. For example, the time when the source cluster middleware is imported is seven ticks, and a timestamp corresponding to the seven ticks is added to the data.
In some embodiments, the starting, in the target cluster, a third process of a corresponding type to import data in the target cluster middleware to a different target component in real time includes: acquiring a timestamp of the data, and judging whether a difference value between a time corresponding to the timestamp and the current time exceeds a threshold value; and responding to the difference value between the time corresponding to the timestamp and the current time exceeding a threshold value, searching for abnormality and giving an alarm. For example, if the current time is seven-point-zero 500 ms, the difference between the current time and the time corresponding to the timestamp is 500 ms, the threshold may be set differently according to the situation, if the threshold is one second, the difference does not exceed the threshold, and if the threshold is 300 ms, the difference exceeds the threshold, and it is necessary to search for an anomaly and perform an alarm.
In some embodiments, initiating a second process between clusters to synchronize data in the source cluster middleware to the target cluster middleware in real time comprises: and adding a second time stamp of the time of importing the middleware of the target cluster into the data.
In some embodiments, the finding an anomaly and alerting comprises: judging whether the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value; and determining that the abnormality is the second process and giving an alarm in response to the fact that the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value. For example, the time when the middleware of the target cluster is imported is seven-point zero 150 milliseconds, and the second threshold is 100 milliseconds, then the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds the second threshold, and it is determined that the exception is the second process and an alarm is given.
In some embodiments, the finding an anomaly and alerting comprises: in response to that the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp does not exceed a second threshold, judging whether the difference between the current time and the time corresponding to the second timestamp exceeds a third threshold; and determining that the abnormality is the third process and giving an alarm in response to the fact that the difference value between the current time and the time corresponding to the second timestamp exceeds a third threshold value. Continuing with the above example, if the time when the middleware of the target cluster is imported is seven-point-zero 50 ms, the second threshold is 100 ms, the current time is seven-point-zero 500 ms, and the third threshold is 200, the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp does not exceed the second threshold, and the difference between the current time and the time corresponding to the second timestamp exceeds the third threshold, it is determined that the exception is the third process, and an alarm is issued.
The monitoring module can be arranged to monitor Kafka logs of the source cluster and the target cluster, print synchronously at the front end in the task execution process, track the progress of each step of the task in the task execution process and perform exception warning.
In the embodiment of the invention, a user can create, start, stop and delete tasks, configure and edit, monitor logs and the like through a front-end module, analyze the submitted tasks, generate configuration files and issue the configuration files to Kafka nodes of all clusters, remotely start Source Connector at a Source cluster to synchronize data of all components to Kafka, further start a Kafka MirrorMaker process to synchronize the Kafka data to Kafka of a target cluster, and start a Sink Connector process at the target cluster to export the Kafka data to all big data components. Migration synchronization of data supports not only between different clusters, but also between different data components of different clusters. The task information is stored in the Mysql metadata base. And the monitoring module prints background logs of each Kafka cluster, tracks task processes in each cluster, and performs monitoring alarm and the like.
It should be particularly noted that, the steps in the embodiments of the cross-cluster real-time data migration and synchronization method described above may be mutually intersected, replaced, added, or deleted, and therefore, these reasonable permutation and combination transformations of the cross-cluster real-time data migration and synchronization method should also belong to the scope of the present invention, and should not limit the scope of the present invention to the embodiments.
In view of the above, a second aspect of the embodiments of the present invention provides a system for migrating and synchronizing data across clusters in real time, including: the analysis module is configured to respond to the received synchronization task, analyze the synchronization task, generate a configuration file, and issue the configuration file to directories corresponding to the source cluster and the target cluster to determine a transmission path; the first transmission module is configured to start a first process of a corresponding type in the source cluster so as to lead data acquired from a source component of the source cluster into a source cluster middleware; the second transmission module is configured to start a second process among the clusters so as to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and a third transmission module, configured to start a third process of a corresponding type in the target cluster to import data in the middleware of the target cluster to different target components in real time.
In some embodiments, the first transmission module is configured to: judging whether the source component and the target component are of the same type; and in response to the source component not being of the same type as the target component, converting data of the source component into a format that is compliant with the target component.
In some embodiments, the first transmission module is configured to: and adding a time stamp of the moment of importing the source cluster middleware into the data.
In some embodiments, the third transmission module is configured to: acquiring a timestamp of the data, and judging whether a difference value between a time corresponding to the timestamp and the current time exceeds a threshold value; and responding to the difference value between the time corresponding to the timestamp and the current time exceeding a threshold value, searching for abnormality and giving an alarm.
In some embodiments, the system further comprises: and the timestamp module is configured to add a second timestamp of the time when the middleware of the target cluster is imported into the data.
In some embodiments, the third transmission module is configured to: judging whether the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value; and determining that the abnormality is the second process and giving an alarm in response to the fact that the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value.
In some embodiments, the third transmission module is configured to: in response to that the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp does not exceed a second threshold, judging whether the difference between the current time and the time corresponding to the second timestamp exceeds a third threshold; and determining that the abnormality is the third process and giving an alarm in response to the fact that the difference value between the current time and the time corresponding to the second timestamp exceeds a third threshold value.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, responding to the received synchronization task, analyzing the synchronization task and generating a configuration file, and issuing the configuration file to the directories corresponding to the source cluster and the target cluster to determine a transmission path; s2, starting a first process of a corresponding type in the source cluster to lead the data acquired from the source component of the source cluster into the middleware of the source cluster; s3, starting a second process among the clusters to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and S4, starting a third process of the corresponding type in the target cluster to lead the data in the middleware of the target cluster to different target components in real time.
In some embodiments, the initiating, in the source cluster, a first process of a corresponding type to import data obtained from a source component of the source cluster into source cluster middleware comprises: judging whether the source component and the target component are of the same type; and in response to the source component not being of the same type as the target component, converting data of the source component into a format that is compliant with the target component.
In some embodiments, the initiating, in the source cluster, a first process of a corresponding type to import data obtained from a source component of the source cluster into source cluster middleware comprises: and adding a time stamp of the moment of importing the source cluster middleware into the data.
In some embodiments, the starting, in the target cluster, a third process of a corresponding type to import data in the target cluster middleware to a different target component in real time includes: acquiring a timestamp of the data, and judging whether a difference value between a time corresponding to the timestamp and the current time exceeds a threshold value; and responding to the difference value between the time corresponding to the timestamp and the current time exceeding a threshold value, searching for abnormality and giving an alarm.
In some embodiments, initiating a second process between clusters to synchronize data in the source cluster middleware to the target cluster middleware in real time comprises: and adding a second time stamp of the time of importing the middleware of the target cluster into the data.
In some embodiments, the finding an anomaly and alerting comprises: judging whether the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value; and determining that the abnormality is the second process and giving an alarm in response to the fact that the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value.
In some embodiments, the finding an anomaly and alerting comprises: in response to that the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp does not exceed a second threshold, judging whether the difference between the current time and the time corresponding to the second timestamp exceeds a third threshold; and determining that the abnormality is the third process and giving an alarm in response to the fact that the difference value between the current time and the time corresponding to the second timestamp exceeds a third threshold value.
Fig. 2 is a schematic hardware structural diagram of an embodiment of the computer device for performing cross-cluster real-time data migration and synchronization according to the present invention.
Taking the apparatus shown in fig. 2 as an example, the apparatus includes a processor 301 and a memory 302, and may further include: an input device 303 and an output device 304.
The processor 301, the memory 302, the input device 303 and the output device 304 may be connected by a bus or other means, and fig. 2 illustrates the connection by a bus as an example.
The memory 302, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the method for cross-cluster real-time data migration and synchronization in the embodiments of the present application. The processor 301 executes various functional applications of the server and data processing by running nonvolatile software programs, instructions and modules stored in the memory 302, that is, implements the method for migrating and synchronizing data across clusters in real time according to the above method embodiments.
The memory 302 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the cross-cluster real-time data migration and synchronization method, and the like. Further, the memory 302 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 302 optionally includes memory located remotely from processor 301, which may be connected to a local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 303 may receive information such as a user name and a password that are input. The output means 304 may comprise a display device such as a display screen.
Program instructions/modules corresponding to one or more cross-cluster real-time data migration and synchronization methods are stored in the memory 302 and, when executed by the processor 301, perform the cross-cluster real-time data migration and synchronization method in any of the above-described method embodiments.
Any embodiment of a computer device executing the cross-cluster real-time data migration and synchronization method described above may achieve the same or similar effects as any of the preceding method embodiments corresponding thereto.
The invention also provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the method as above.
Finally, it should be noted that, as one of ordinary skill in the art can appreciate that all or part of the processes of the methods of the above embodiments can be implemented by a computer program to instruct related hardware, and the program of the method for real-time data migration and synchronization across clusters can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method for cross-cluster real-time data migration and synchronization, comprising the steps of:
in response to receiving a synchronization task, analyzing the synchronization task and generating a configuration file, and issuing the configuration file to directories corresponding to a source cluster and a target cluster to determine a transmission path;
starting a first process of a corresponding type in the source cluster to lead data acquired from a source component of the source cluster into a source cluster middleware;
starting a second process among the clusters to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and
and starting a third process of a corresponding type in the target cluster to lead the data in the middleware of the target cluster to different target components in real time.
2. The method of claim 1, wherein the initiating a first process of a corresponding type in the source cluster to import data obtained from a source component of the source cluster into source cluster middleware comprises:
judging whether the source component and the target component are of the same type; and
in response to the source component not being of the same type as the target component, converting data of the source component into a format that is compliant with the target component.
3. The method of claim 1, wherein the initiating a first process of a corresponding type in the source cluster to import data obtained from a source component of the source cluster into source cluster middleware comprises:
and adding a time stamp of the moment of importing the source cluster middleware into the data.
4. The method of claim 3, wherein the initiating a third process of a corresponding type in the target cluster to import data in the target cluster middleware to a different target component in real time comprises:
acquiring a timestamp of the data, and judging whether a difference value between a time corresponding to the timestamp and the current time exceeds a threshold value; and
and in response to the fact that the difference value between the time corresponding to the timestamp and the current time exceeds a threshold value, searching for abnormality and giving an alarm.
5. The method of claim 4, wherein the initiating a second inter-cluster process to synchronize data in the source cluster middleware to the target cluster middleware in real-time comprises:
and adding a second time stamp of the time of importing the middleware of the target cluster into the data.
6. The method of claim 5, wherein the locating anomalies and alerting comprises:
judging whether the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value; and
and determining that the abnormality is the second process and giving an alarm in response to the fact that the difference value between the time corresponding to the second timestamp and the time corresponding to the timestamp exceeds a second threshold value.
7. The method of claim 6, wherein the locating anomalies and alerting comprises:
in response to that the difference between the time corresponding to the second timestamp and the time corresponding to the timestamp does not exceed a second threshold, judging whether the difference between the current time and the time corresponding to the second timestamp exceeds a third threshold; and
and determining that the abnormality is the third process and giving an alarm in response to the fact that the difference value between the current time and the time corresponding to the second timestamp exceeds a third threshold value.
8. A method for cross-cluster real-time data migration and synchronization, comprising:
the analysis module is configured to respond to the received synchronization task, analyze the synchronization task, generate a configuration file, and issue the configuration file to directories corresponding to the source cluster and the target cluster to determine a transmission path;
the first transmission module is configured to start a first process of a corresponding type in the source cluster so as to lead data acquired from a source component of the source cluster into a source cluster middleware;
the second transmission module is configured to start a second process among the clusters so as to synchronize the data in the source cluster middleware to the target cluster middleware in real time; and
and the third transmission module is configured to start a third process of a corresponding type in the target cluster so as to import the data in the middleware of the target cluster into different target components in real time.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011231345.XA CN112463762A (en) | 2020-11-06 | 2020-11-06 | Method, system, device and medium for cross-cluster real-time data migration and synchronization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011231345.XA CN112463762A (en) | 2020-11-06 | 2020-11-06 | Method, system, device and medium for cross-cluster real-time data migration and synchronization |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112463762A true CN112463762A (en) | 2021-03-09 |
Family
ID=74825869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011231345.XA Withdrawn CN112463762A (en) | 2020-11-06 | 2020-11-06 | Method, system, device and medium for cross-cluster real-time data migration and synchronization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463762A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108628874A (en) * | 2017-03-17 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data |
| WO2019154394A1 (en) * | 2018-02-12 | 2019-08-15 | 中兴通讯股份有限公司 | Distributed database cluster system, data synchronization method and storage medium |
| CN110334075A (en) * | 2019-04-04 | 2019-10-15 | 平安科技（深圳）有限公司 | Data migration method and relevant device based on message-oriented middleware |
-
2020
- 2020-11-06 CN CN202011231345.XA patent/CN112463762A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108628874A (en) * | 2017-03-17 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data |
| WO2019154394A1 (en) * | 2018-02-12 | 2019-08-15 | 中兴通讯股份有限公司 | Distributed database cluster system, data synchronization method and storage medium |
| CN110334075A (en) * | 2019-04-04 | 2019-10-15 | 平安科技（深圳）有限公司 | Data migration method and relevant device based on message-oriented middleware |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109918349B (en) | Log processing method, log processing device, storage medium and electronic device | |
| CN107368503B (en) | Data synchronization method and system based on button | |
| CN110309161B (en) | Data synchronization method and device and server | |
| CN111752799A (en) | Service link tracking method, device, equipment and storage medium | |
| CN109144785B (en) | Method and apparatus for backing up data | |
| CN110971655A (en) | Offline client playback and synchronization | |
| CN107766509B (en) | Method and device for static backup of webpage | |
| WO2014049804A1 (en) | System operation trace method in distributed system | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| US10318385B2 (en) | Service recovery using snapshots and interservice messages | |
| CN106682210B (en) | Log file query method and device | |
| CN109299116B (en) | Data synchronization method, device, equipment and readable storage medium | |
| CN106649000A (en) | Fault recovery method for real-time processing engine, and corresponding server | |
| US10129328B2 (en) | Centralized management of webservice resources in an enterprise | |
| CN112579406B (en) | Log call chain generation method and device | |
| KR102031589B1 (en) | Methods and systems for processing relationship chains, and storage media | |
| CN110941658A (en) | Data export method, device, server and storage medium | |
| WO2023240995A1 (en) | Data recovery method and apparatus for dual-machine hot standby system, and medium | |
| CN112463762A (en) | Method, system, device and medium for cross-cluster real-time data migration and synchronization | |
| CN115426216A (en) | Internet of things node interconnection method based on complex virtual instrument software | |
| CN115292307A (en) | Data synchronization system, method and corresponding computer equipment and storage medium | |
| CN104834724B (en) | A kind of database synchronization method and device | |
| CN114281600A (en) | Disaster recovery backup and recovery method, device, equipment and storage medium | |
| CN110515803B (en) | Processing method and device for log message and electronic equipment | |
| CN111596933A (en) | File processing method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210309 |