CN117786001A - Data synchronization method, device, electronic equipment and storage medium - Google Patents
Data synchronization method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117786001A CN117786001A CN202311632464.XA CN202311632464A CN117786001A CN 117786001 A CN117786001 A CN 117786001A CN 202311632464 A CN202311632464 A CN 202311632464A CN 117786001 A CN117786001 A CN 117786001A
- Authority
- CN
- China
- Prior art keywords
- data
- cluster
- message middleware
- target
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 230000001360 synchronised effect Effects 0.000 claims abstract description 50
- 230000008859 change Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 6
- 238000013524 data verification Methods 0.000 claims description 5
- 230000000977 initiatory effect Effects 0.000 claims description 5
- 230000001502 supplementing effect Effects 0.000 claims description 4
- 230000003139 buffering effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 10
- 230000006399 behavior Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000000047 product Substances 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Abstract
The invention discloses a data synchronization method, a data synchronization device, electronic equipment and a storage medium. In the embodiment of the invention, the incremental data generated by all operations are firstly cached in the message middleware, and after the stock service data in the source cluster is synchronized to the new target cluster, the incremental data cached in the message middleware is synchronized to the target cluster, so that the loss of the incremental data is avoided, and the integrity of data synchronization is ensured.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data synchronization method, a data synchronization device, electronic equipment and a storage medium.
Background
Currently, in the case that no behavior operation (such as an operation of adding or modifying service data in a source cluster) occurs by a user, the full data can be directly synchronized from the source cluster to a new target cluster. Under the condition that some behavior operations occur to a user, synchronizing full-quantity data (including stock data and incremental data) from a source cluster to a new target cluster can be divided into the following steps of performing full-quantity data synchronization, namely firstly enabling a service party to start a double-writing function, and triggering a server to bidirectionally write the incremental data generated by the behavior operations after some behavior operations occur to the user, and writing the incremental data into the source cluster and the new target cluster respectively; the stock data is then synchronized from the source cluster to the new target cluster.
However, the double write mechanism is started to synchronize data, which is easy to cause data loss.
Disclosure of Invention
The embodiment of the invention provides a data synchronization method, a data synchronization device, electronic equipment and a storage medium, which are used for guaranteeing the integrity of data synchronization.
In a first aspect, an embodiment of the present invention provides a data synchronization method, including:
in the data synchronization task, responding to the change operation of service data of a target service deployed in a source cluster, caching the generated incremental service data to an external message middleware, wherein the message middleware comprises an incremental consumption switch, and the incremental consumption switch is used for controlling the data circulation between the message middleware and a new target cluster; under the condition that no data is updated in the message middleware within a preset time, responding to the starting operation of an incremental consumption switch of the message middleware, and synchronizing the changed business data cached in the message middleware into the target cluster; and synchronizing the stock service data in the source cluster to the target cluster to complete the data synchronization task.
Optionally, in response to a change operation on service data of a target service deployed in the source cluster, caching the generated incremental service data to an external message middleware, and further including: responding to the data synchronization instruction, and determining an ES source cluster; querying a client associated with the ES source cluster according to the ES source cluster; and initiating an instruction for caching the incremental data to the message middleware to the client, and controlling the client to cache the incremental data of the ES source cluster to the message middleware.
Optionally, synchronizing the stock service data in the source cluster in the target cluster includes: acquiring a directory file corresponding to the stock service data to be synchronized, and scanning the directory file corresponding to the stock service data to be synchronized according to the directory file; performing target file configuration in target cluster creation based on the scanning file; and synchronizing the directory files corresponding to the stock service data to be synchronized according to the configured target files.
Optionally, obtaining a target file corresponding to the stock service data to be synchronized includes: acquiring task requirements of the data synchronization task, wherein the task requirements comprise a time range of the data synchronization task; traversing the time range endogenous cluster file catalogue of the data synchronization task, and taking the data in the time range of the data synchronization task as stock data.
Optionally, synchronizing the directory file corresponding to the stock service data to be synchronized according to the configured target file includes: writing indexes corresponding to all to-be-synchronized stock service data in the source cluster into a preset synchronization queue; determining the number of parallel tasks and the synchronization speed of parallel synchronization according to the configured target file; and starting a data synchronization process corresponding to the number of parallel tasks, and respectively transmitting indexes to be synchronized corresponding to the number of parallel tasks to the target cluster according to the synchronization speed.
Optionally, the method further comprises: after the data synchronization task is completed, synchronous data verification is carried out according to a preset rule, whether the sum of the directory files of the source cluster and the directory files cached in the message middleware is consistent with the directory files in the target cluster after data synchronization within a preset time is judged, and a synchronization completion signal is output under the condition that the judgment result is consistent.
Optionally, the method further comprises: and under the condition that the judging result is inconsistent, supplementing the directory files in the target cluster according to the directory files of the source cluster and the directory files cached in the message middleware in the preset time so as to ensure the consistency of the sum of the directory files of the source cluster and the directory files cached in the message middleware in the preset time and the directory files in the target cluster after data synchronization.
In a second aspect, an embodiment of the present invention provides a data synchronization apparatus, including: the data caching module is used for responding to the change operation of the service data of the target service deployed in the source cluster in the data synchronization task and caching the generated incremental service data to an external message middleware, wherein the message middleware comprises an incremental consumption switch, and the incremental consumption switch is used for controlling the data circulation between the message middleware and the new target cluster; the first data synchronization module is used for responding to the starting operation of an incremental consumption switch of the message middleware under the condition that no data is updated in the message middleware within preset time, and synchronizing the changed business data cached in the message middleware into the target cluster; and the second data synchronization module is used for synchronizing the stock service data in the source cluster to the target cluster so as to complete the data synchronization task.
In a third aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory for storing one or more computer instructions which when executed by the processor perform the steps in the above method.
In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program which, when executed by one or more processors, causes the one or more processors to perform the steps in the above method.
In the embodiment of the invention, the incremental data generated by all operations are firstly cached in the message middleware, and after the stock service data in the source cluster is synchronized to the new target cluster, the incremental data cached in the message middleware is synchronized to the target cluster, so that the loss of the incremental data is avoided, and the integrity of data synchronization is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief description will be given below of the drawings required for the embodiments or the prior art descriptions, and it is obvious that the drawings in the following description are some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a data synchronization method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data synchronization device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device corresponding to the data synchronization device provided in the embodiment shown in fig. 2.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, the "plurality" generally includes at least two.
It should be understood that the term "and/or" as used herein is merely one relationship describing the association of the associated objects, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a product or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such product or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a commodity or system comprising such elements.
In the prior art, when a user performs some behavior operations (operations such as adding, deleting or modifying) on a service in a source cluster, and needs to synchronize data in the source cluster to other target clusters, service double writing is started, namely when the user performs some behavior operations on the service in the source cluster, incremental data generated by the behavior operations are synchronously written into the source cluster and the target cluster, and then stock data in the source cluster is synchronized into the target cluster. However, the function of dual writing of the service is started, on the one hand, if one service A exists in the source cluster and no service A exists in the target cluster, when a user performs, for example, modification operation on the service A, only data for performing modification operation on the A can be written in the source cluster, and if no service A exists in the target cluster, service data for performing modification operation on the A cannot be written in the target cluster, namely, writing in the target cluster fails, which is equivalent to loss of incremental data in the target cluster. On the other hand, when double writing is started, compared with when double writing is not started, one implementation step is added, and delay is caused, and the delay can also cause the loss of incremental data.
Fig. 1 is a flow chart of a data synchronization method according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
101. in the data synchronization task, responding to the change operation of service data of a target service deployed in a source cluster, caching the generated incremental service data to an external message middleware, wherein the message middleware comprises an incremental consumption switch, and the incremental consumption switch is used for controlling the data circulation between the message middleware and a new target cluster;
102. under the condition that no data is updated in the message middleware within a preset time, responding to the starting operation of an incremental consumption switch of the message middleware, and synchronizing the changed business data cached in the message middleware into a target cluster;
103. and synchronizing the stock service data in the source cluster to the target cluster to complete the data synchronization task.
In this embodiment, the target service may be a driving service, a dispatch service, a cleaning service, or the like. The incremental business data is newly added business data generated by changing the operation of the pointer to the target business. The change operation includes, but is not limited to: and performing operations such as adding, modifying and deleting on the target service. The message middleware may be Windows middleware such as IIS, ASP.NET, sharePoint, bizTalk, etc. The message middleware may be Linux middleware, such as nginx, tomcat, memcached, etc. Cross-platform middleware such as Java EE, zeromq, etc. Open source middleware such as AMQ, activeMQ, rabbitMQ and the like.
In some embodiments, in response to a change operation to service data of a target service deployed in a source cluster, caching the generated incremental service data to an external message middleware, further comprising: responding to the data synchronization instruction, and determining an ES source cluster; querying a client associated with the ES source cluster according to the ES source cluster; and initiating an instruction for caching the incremental data to the message middleware to the client, and controlling the client to cache the incremental data of the ES source cluster to the message middleware. The ES source cluster is a cluster for storing stock data of the target service, wherein the stock data is original service data without changing the target service.
In some embodiments, synchronizing the stock traffic data in the source cluster to the target cluster includes: acquiring a directory file corresponding to the stock service data to be synchronized, and scanning the directory file corresponding to the stock service data to be synchronized according to the scanned file of the directory file corresponding to the stock service data to be synchronized; performing target file configuration in target cluster creation based on the scanned file; and synchronizing the directory files corresponding to the stock service data to be synchronized according to the configured target files.
Optionally, obtaining a target file corresponding to the stock service data to be synchronized includes: acquiring task requirements of a data synchronization task, wherein the task requirements comprise a time range of the data synchronization task; traversing the source cluster file catalogue in the time range of the data synchronization task, and taking the data in the time range of the data synchronization task as stock data. The time range may be a set time period before the current time, such as 1 minute before the current time.
In some embodiments, synchronizing, according to the configured target file, a directory file corresponding to the stock service data to be synchronized includes: writing indexes corresponding to all the stock service data to be synchronized in the source cluster into a preset synchronization queue; determining the number of parallel tasks and the synchronization speed of parallel synchronization according to the configured target file; and starting a data synchronization process corresponding to the number of parallel tasks, and respectively transmitting indexes to be synchronized corresponding to the number of parallel tasks to the target cluster according to the synchronization speed.
Further optionally, after completing the data synchronization task, performing synchronization data verification according to a preset rule, judging whether the sum of the directory files of the source cluster and the directory files cached in the message middleware is consistent with the directory files in the target cluster after data synchronization within a preset time, and outputting a synchronization completion signal if the judgment result is consistent.
Further optionally: and under the condition that the judging result is inconsistent, supplementing the directory files in the target cluster according to the directory files of the source cluster and the directory files cached in the message middleware in the preset time so as to ensure the consistency of the sum of the directory files of the source cluster and the directory files cached in the message middleware in the preset time and the directory files in the target cluster after the data synchronization.
In the embodiment of the invention, the incremental data generated by all operations are firstly cached in the message middleware, and after the stock service data in the source cluster is synchronized to the new target cluster, the incremental data cached in the message middleware is synchronized to the target cluster, so that the loss of the incremental data is avoided, and the integrity of data synchronization is ensured.
Based on the same concept, the embodiment of the present application further provides a data synchronization device, and fig. 2 is a schematic structural diagram of the data synchronization device provided in the embodiment of the present application. The device comprises:
the data caching module 21 is configured to cache, in response to a change operation of service data of a target service deployed in a source cluster, generated incremental service data to an external message middleware, where the message middleware includes an incremental consumption switch, and the incremental consumption switch is configured to control data flow between the message middleware and a new target cluster;
the first data synchronization module 22 is configured to synchronize, in response to an on operation of an incremental consumption switch of the message middleware, the changed service data cached in the message middleware to the target cluster in the case that no data update is performed in the message middleware within a preset time; the method comprises the steps of,
and the second data synchronization module 23 is configured to synchronize the stock service data in the source cluster to the target cluster, so as to complete the data synchronization task.
In an alternative embodiment, the first data synchronization module 22 is specifically configured to, when responding to a change operation on service data of a target service deployed in a source cluster, cache generated incremental service data to an external message middleware: responding to the data synchronization instruction, and determining an ES source cluster; querying a client associated with the ES source cluster according to the ES source cluster; and initiating an instruction for caching the incremental data to the message middleware to the client, and controlling the client to cache the incremental data of the ES source cluster to the message middleware.
In an alternative embodiment, the second data synchronization module 23 is specifically configured to, when configured to synchronize the stock traffic data in the source cluster to the target cluster: acquiring a directory file corresponding to the stock service data to be synchronized, and scanning the directory file corresponding to the stock service data to be synchronized according to the scanned file of the directory file corresponding to the stock service data to be synchronized; performing target file configuration in target cluster creation based on the scanned file; and synchronizing the directory files corresponding to the stock service data to be synchronized according to the configured target files.
Optionally, the second data synchronization module 23 is specifically configured to, when configured to obtain a target file corresponding to the stock service data to be synchronized: acquiring task requirements of a data synchronization task, wherein the task requirements comprise a time range of the data synchronization task; traversing the source cluster file catalogue in the time range of the data synchronization task, and taking the data in the time range of the data synchronization task as stock data.
Optionally, the second data synchronization module 23 is specifically configured to, when configured to synchronize, according to the configured target file, a directory file corresponding to the stock service data to be synchronized: writing indexes corresponding to all the stock service data to be synchronized in the source cluster into a preset synchronization queue; determining the number of parallel tasks and the synchronization speed of parallel synchronization according to the configured target file; and starting a data synchronization process corresponding to the number of parallel tasks, and respectively transmitting indexes to be synchronized corresponding to the number of parallel tasks to the target cluster according to the synchronization speed.
Further optionally, the method further comprises: and the data judging module 24 is used for carrying out synchronous data verification according to a preset rule after completing the data synchronization task, judging whether the sum of the directory files of the source cluster and the directory files cached in the message middleware is consistent with the directory files in the target cluster after carrying out data synchronization in a preset time, and outputting a synchronization completion signal under the condition that the judging result is consistent.
Further optionally, the method further comprises: the data supplementing module 25 is configured to supplement the directory files in the target cluster according to the directory files of the source cluster and the directory files cached in the message middleware in a preset time when the judging result is inconsistent, so as to ensure consistency between the sum of the directory files of the source cluster and the directory files cached in the message middleware in the preset time and the directory files in the target cluster after data synchronization.
It should be noted that, the specific implementation manner of executing each step by each module in this embodiment may be referred to the detailed description in the method embodiment, and will not be repeated here.
In the embodiments of the present application, the incremental data generated by all operations are cached in the message middleware, and after the stock service data in the source cluster is synchronized to the new target cluster, the incremental data cached in the message middleware is synchronized to the target cluster, so that the loss of the incremental data is avoided, and the integrity of data synchronization is ensured.
In one possible design, the structure of the data processing apparatus shown in fig. 2 may be implemented as an electronic device, and as shown in fig. 3, a schematic structural diagram of an electronic device corresponding to the data synchronization apparatus provided in the embodiment of fig. 2 may include: a processor 31, a memory 32, the memory 32 for storing one or more computer instructions, wherein the one or more computer instructions when executed by the processor 31 perform the steps of:
in the data synchronization task, responding to the change operation of service data of a target service deployed in a source cluster, caching the generated incremental service data to an external message middleware, wherein the message middleware comprises an incremental consumption switch, and the incremental consumption switch is used for controlling the data circulation between the message middleware and a new target cluster; under the condition that no data is updated in the message middleware within a preset time, responding to the starting operation of an incremental consumption switch of the message middleware, and synchronizing the changed business data cached in the message middleware into a target cluster; and synchronizing the stock service data in the source cluster to the target cluster to complete the data synchronization task.
In an alternative embodiment, the processor 31 is specifically configured to, when configured to cache the generated incremental service data to the external message middleware in response to a change operation on the service data of the target service deployed in the source cluster: responding to the data synchronization instruction, and determining an ES source cluster; querying a client associated with the ES source cluster according to the ES source cluster; and initiating an instruction for caching the incremental data to the message middleware to the client, and controlling the client to cache the incremental data of the ES source cluster to the message middleware.
In an alternative embodiment, the processor 31 is configured, when configured to synchronize the stock traffic data in the source cluster to the target cluster, to: acquiring a directory file corresponding to the stock service data to be synchronized, and scanning the directory file corresponding to the stock service data to be synchronized according to the scanned file of the directory file corresponding to the stock service data to be synchronized; performing target file configuration in target cluster creation based on the scanned file; and synchronizing the directory files corresponding to the stock service data to be synchronized according to the configured target files.
Optionally, the processor 31 is specifically configured to, when configured to obtain a target file corresponding to the stock service data to be synchronized: acquiring task requirements of a data synchronization task, wherein the task requirements comprise a time range of the data synchronization task; traversing the source cluster file catalogue in the time range of the data synchronization task, and taking the data in the time range of the data synchronization task as stock data.
Optionally, when the processor 31 is configured to synchronize, according to the configured target file, the directory file corresponding to the stock service data to be synchronized, the processor is specifically configured to: writing indexes corresponding to all the stock service data to be synchronized in the source cluster into a preset synchronization queue; determining the number of parallel tasks and the synchronization speed of parallel synchronization according to the configured target file; and starting a data synchronization process corresponding to the number of parallel tasks, and respectively transmitting indexes to be synchronized corresponding to the number of parallel tasks to the target cluster according to the synchronization speed.
Further optionally, the processor 31 is further configured to perform synchronization data verification according to a preset rule after completing the data synchronization task, determine whether the sum of the directory file of the source cluster and the directory file cached in the message middleware within a preset time is consistent with the directory file in the target cluster after performing data synchronization, and output a synchronization completion signal if the determination result is consistent.
Further optionally, the processor 31 is further configured to supplement the directory file in the target cluster according to the directory file of the source cluster and the directory file cached in the message middleware in a preset time, so as to ensure consistency between the sum of the directory file of the source cluster and the directory file cached in the message middleware in the preset time and the directory file in the target cluster after data synchronization, if the judging result is inconsistent.
Optionally, a communication interface 33 may also be included in the electronic device for communicating with other devices.
It should be noted that, the specific implementation manner of executing each step by each module in this embodiment may be referred to the detailed description in the method embodiment, and will not be repeated here.
In addition, an embodiment of the present invention provides a computer storage medium storing a computer program, where the computer program makes a client implement the data processing method in the embodiment shown in fig. 1 when the computer program is executed.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by adding necessary general purpose hardware platforms, or may be implemented by a combination of hardware and software. Based on such understanding, the foregoing aspects, in essence and portions contributing to the art, may be embodied in the form of a computer program product, which may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable resource updating apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable resource updating apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable resource updating device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable resource updating apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of data synchronization, comprising:
in the data synchronization task, responding to the change operation of service data of a target service deployed in a source cluster, caching the generated incremental service data to an external message middleware, wherein the message middleware comprises an incremental consumption switch, and the incremental consumption switch is used for controlling the data circulation between the message middleware and a new target cluster;
under the condition that no data is updated in the message middleware within a preset time, responding to the starting operation of an incremental consumption switch of the message middleware, and synchronizing the changed business data cached in the message middleware into the target cluster; the method comprises the steps of,
and synchronizing the stock service data in the source cluster to the target cluster to complete the data synchronization task.
2. The method of claim 1, wherein buffering the generated incremental traffic data to the external message middleware in response to a change operation to traffic data of a target traffic deployed in the source cluster, comprises:
responding to the data synchronization instruction, and determining an ES source cluster;
querying a client associated with the ES source cluster according to the ES source cluster;
and initiating an instruction for caching the incremental data to the message middleware to the client, and controlling the client to cache the incremental data of the ES source cluster to the message middleware.
3. The method of claim 1, wherein synchronizing the inventory traffic data in the source cluster in the target cluster comprises:
acquiring a directory file corresponding to the stock service data to be synchronized, and scanning the directory file corresponding to the stock service data to be synchronized according to the directory file; the method comprises the steps of,
performing target file configuration in target cluster creation based on the scanning file;
and synchronizing the directory files corresponding to the stock service data to be synchronized according to the configured target files.
4. A method according to claim 3, wherein obtaining a target file corresponding to stock service data to be synchronized comprises:
acquiring task requirements of the data synchronization task, wherein the task requirements comprise a time range of the data synchronization task;
traversing the time range endogenous cluster file catalogue of the data synchronization task, and taking the data in the time range of the data synchronization task as stock data.
5. The method of claim 3, wherein synchronizing the directory file corresponding to the inventory service data to be synchronized according to the configured target file comprises:
writing indexes corresponding to all to-be-synchronized stock service data in the source cluster into a preset synchronization queue;
determining the number of parallel tasks and the synchronization speed of parallel synchronization according to the configured target file;
and starting a data synchronization process corresponding to the number of parallel tasks, and respectively transmitting indexes to be synchronized corresponding to the number of parallel tasks to the target cluster according to the synchronization speed.
6. The method as recited in claim 2, further comprising:
after the data synchronization task is completed, synchronous data verification is carried out according to a preset rule, whether the sum of the directory files of the source cluster and the directory files cached in the message middleware is consistent with the directory files in the target cluster after data synchronization within a preset time is judged, and a synchronization completion signal is output under the condition that the judgment result is consistent.
7. The method as recited in claim 6, further comprising:
and under the condition that the judging result is inconsistent, supplementing the directory files in the target cluster according to the directory files of the source cluster and the directory files cached in the message middleware in the preset time so as to ensure the consistency of the sum of the directory files of the source cluster and the directory files cached in the message middleware in the preset time and the directory files in the target cluster after data synchronization.
8. A data synchronization device, comprising:
the data caching module is used for responding to the change operation of the service data of the target service deployed in the source cluster in the data synchronization task and caching the generated incremental service data to an external message middleware, wherein the message middleware comprises an incremental consumption switch, and the incremental consumption switch is used for controlling the data circulation between the message middleware and the new target cluster;
the first data synchronization module is used for responding to the starting operation of an incremental consumption switch of the message middleware under the condition that no data is updated in the message middleware within preset time, and synchronizing the changed business data cached in the message middleware into the target cluster; the method comprises the steps of,
and the second data synchronization module is used for synchronizing the stock service data in the source cluster to the target cluster so as to complete the data synchronization task.
9. An electronic device, comprising: a processor, a memory for storing one or more computer instructions, wherein the one or more computer instructions when executed by the processor implement the method of any of claims 1-7.
10. A computer readable storage medium storing a computer program, characterized in that the computer program, when executed by one or more processors, causes the one or more processors to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311632464.XA CN117786001A (en) | 2023-11-30 | 2023-11-30 | Data synchronization method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311632464.XA CN117786001A (en) | 2023-11-30 | 2023-11-30 | Data synchronization method, device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117786001A true CN117786001A (en) | 2024-03-29 |
Family
ID=90395260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311632464.XA Pending CN117786001A (en) | 2023-11-30 | 2023-11-30 | Data synchronization method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117786001A (en) |
-
2023
- 2023-11-30 CN CN202311632464.XA patent/CN117786001A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11221995B2 (en) | Data replication from a cloud-based storage resource | |
| CN108628688B (en) | Message processing method, device and equipment | |
| CN109032507B (en) | Method, system and related assembly for generating source volume storage snapshot | |
| US10609141B2 (en) | Methods and apparatuses for cluster switching | |
| CN113158110A (en) | Data processing method and equipment | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| CN112235132B (en) | Method, device, medium and server for dynamically configuring service | |
| CN110244951B (en) | Application publishing method and device | |
| CN112597151A (en) | Data processing method, device, equipment and storage medium | |
| CN111142975B (en) | State machine persistence method and state machine persistence system | |
| CN112363980A (en) | Data processing method and device for distributed system | |
| CN117786001A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN110764930A (en) | Request or response processing method and device based on message mode | |
| CN115586872A (en) | Container mirror image management method, device, equipment and storage medium | |
| CN113434476B (en) | Data synchronization method, device, equipment, system, storage medium and program product | |
| CN113656496A (en) | Data processing method and system | |
| CN110688057B (en) | Distributed storage method and device | |
| CN113127438A (en) | Method, apparatus, server and medium for storing data | |
| CN113297267A (en) | Data caching and task processing method, device, equipment and storage medium | |
| CN112579117A (en) | Data upgrading method and device | |
| CN110716699A (en) | Method and apparatus for writing data | |
| CN110933120A (en) | Computing method, device and system based on hybrid cloud | |
| CN115604290B (en) | Kafka message execution method, device, equipment and storage medium | |
| CN116467270A (en) | Data management system, data updating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |