CN118250289A

CN118250289A - Data synchronization method, device, equipment and storage medium

Info

Publication number: CN118250289A
Application number: CN202211656765.1A
Authority: CN
Inventors: 黄群; 董俊峰; 强群力; 刘超千; 赵彤; 沈村敬
Original assignee: NetsUnion Clearing Corp
Current assignee: NetsUnion Clearing Corp
Priority date: 2022-12-22
Filing date: 2022-12-22
Publication date: 2024-06-25

Abstract

The embodiment of the specification provides a data synchronization method, a device, equipment and a storage medium, which are applied to data synchronization middleware, wherein the data synchronization middleware is used for synchronizing data between a first data source and a second data source, and the second data source is a backup data source during the unavailable period of the first data source, and the method comprises the following steps: acquiring first data to be synchronized generated by the second data source in the unavailable period, and executing a synchronization operation of synchronizing the first data to be synchronized to the first data source after the first data source is changed to an available state; and in the process of synchronizing the first data to be synchronized, acquiring second data to be synchronized generated by the first data source in an available state, and executing the synchronization operation of synchronizing the second data to be synchronized to the second data source. By the embodiment, bidirectional data synchronization can be realized, and increasingly complex data synchronization requirements are met.

Description

Data synchronization method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data synchronization method, apparatus, device, and storage medium.

Background

In the field of data processing, data in one data source is often synchronized to another data source using data synchronization techniques. At present, in the process of synchronizing data in one data source to another data source, only one-way synchronization of the data can be realized, and two-way synchronization of the data cannot be realized. For example, in the data synchronization process, only data can be synchronized from a designated data source a to a designated data source B, and in the synchronization process, data in the data source B cannot be synchronized to the data source a any more. Therefore, the data synchronization direction of the existing data synchronization technology is relatively fixed, bidirectional synchronization cannot be realized, and increasingly complex data synchronization requirements cannot be met.

Disclosure of Invention

An embodiment of the present disclosure provides a data synchronization method, apparatus, device, and storage medium, which can realize bidirectional synchronization of data, and meet increasingly complex data synchronization requirements.

To achieve the above object, an embodiment of the present disclosure provides a data synchronization method applied to a data synchronization middleware, where the data synchronization middleware is configured to synchronize data between a first data source and a second data source, and the second data source is a backup data source during a period when the first data source is unavailable, the method includes:

acquiring first data to be synchronized generated by the second data source in the unavailable period, and executing a synchronization operation of synchronizing the first data to be synchronized to the first data source after the first data source is changed to an available state;

And in the process of synchronizing the first data to be synchronized, acquiring second data to be synchronized generated by the first data source in an available state, and executing the synchronization operation of synchronizing the second data to be synchronized to the second data source.

Another embodiment of the present disclosure provides a data synchronization apparatus applied to a data synchronization middleware for synchronizing data between a first data source and a second data source, the second data source being a backup data source during a period when the first data source is unavailable, the apparatus comprising:

The first synchronization module is used for acquiring first data to be synchronized generated by the second data source in the unavailable period, and executing a synchronization operation of synchronizing the first data to be synchronized to the first data source after the first data source is changed to an available state;

And the second synchronization module is used for acquiring second data to be synchronized generated by the first data source in an available state in the process of synchronizing the first data to be synchronized, and executing the synchronization operation of synchronizing the second data to be synchronized to the second data source.

Another embodiment of the present specification provides a data synchronization apparatus for synchronizing data between a first data source and a second data source, the second data source being a backup data source during a period when the first data source is unavailable, the apparatus comprising:

A processor; and

A memory arranged to store computer executable instructions that, when executed, cause the processor to:

Another embodiment of the present specification provides a storage medium for use in a data synchronization middleware for synchronizing data between a first data source and a second data source, the second data source being a backup data source during periods when the first data source is unavailable, the storage medium for storing computer executable instructions that, when executed by a processor, implement the following:

In the data synchronization method, the device, the equipment and the storage medium provided by one or more embodiments of the present disclosure, during the unavailable period of the first data source, the second data source is a backup data source of the second data source, the first data to be synchronized generated by the second data source during the unavailable period of the first data source is obtained, after the first data source is available, the synchronization operation of synchronizing the first data to the first data source is performed, and during the process of synchronizing the first data to be synchronized, the second data to be synchronized generated by the first data source in the available state is obtained, and the synchronization operation of synchronizing the second data to the second data source is performed, so that bidirectional synchronization of data can be realized, and increasingly complex data synchronization requirements are satisfied.

Drawings

In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of an application scenario of a data synchronization method according to an embodiment of the present disclosure;

FIG. 2 is a flow chart of a data synchronization method according to an embodiment of the present disclosure;

FIG. 3a is a schematic diagram of data synchronization according to an embodiment of the present disclosure;

FIG. 3b is a schematic diagram of data synchronization according to another embodiment of the present disclosure;

FIG. 3c is a schematic diagram of data synchronization according to another embodiment of the present disclosure;

FIG. 4a is a schematic diagram of data synchronization according to another embodiment of the present disclosure;

FIG. 4b is a schematic diagram of data synchronization according to another embodiment of the present disclosure;

FIG. 5 is a schematic diagram illustrating a module composition of a data synchronization device according to an embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of a data synchronization device according to an embodiment of the present disclosure.

Detailed Description

In order to make the technical solutions in this document better understood by those skilled in the art, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the drawings in the embodiments of this specification, and it is apparent that the described embodiments are only some embodiments of this document, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

Fig. 1 is a schematic application scenario of a data synchronization method according to an embodiment of the present disclosure, as shown in fig. 1, the first data source may be any form of data source, such as a database or a distributed data storage cluster, and the second data source may be any form of data source, such as a database or a distributed data storage cluster. In one example, the first data source and the second data source may each be a GlusterFS data storage cluster. The data synchronization middleware may be a data synchronizer for synchronizing data between the first data source and the second data source.

In one case, the first data source and the second data source provide data storage services for different users of the same service, for example, the first data source is a distributed data storage cluster in a beijing machine room of the payment service, provides data storage services for beijing users of the payment service, stores data such as bills of the beijing users, and the second data source is a distributed data storage cluster in an Shanghai machine room of the payment service, provides data storage services for Shanghai users of the payment service, and stores data such as bills of Shanghai users. The second data source is used as a backup data source of the first data source during the unavailable period of the first data source, and provides data storage services for both the user corresponding to the first data source and the user corresponding to the second data source.

In another case, the first data source is a cold backup data source of the second data source, when the first data source is in an available state, the first data source is preferentially used for data storage, and when the first data source is unavailable, the second data source is used for data storage. For example, the first data source provides data storage service for Beijing users of the payment service, and when the first data source is unavailable, the second data source replaces the first data source to provide data storage service for the Beijing users of the payment service.

The first data source is not available, which may be that the first data source cannot store data any more due to a failure of a server in the first data source, or that the first data source cannot be continuously invoked due to a failure of a service layer corresponding to the first data source.

The data synchronization middleware can execute the data synchronization method in the embodiment, can acquire first data to be synchronized generated by the second data source in the unavailable period of the first data source, execute the synchronization operation of synchronizing the first data to the first data source after the first data source is changed to be in an available state, and acquire second data to be synchronized generated by the first data source in the available state in the process of synchronizing the first data to be synchronized, and execute the synchronization operation of synchronizing the second data to be synchronized to the second data source.

It can be seen that, by the method in this embodiment, during the unavailable period of the first data source, the second data source is a backup data source of the second data source, so as to obtain the first data to be synchronized generated by the second data source during the unavailable period of the first data source, after the first data source is available, perform a synchronization operation of synchronizing the first data to the first data source, and during the process of synchronizing the first data to be synchronized, obtain the second data to be synchronized generated by the first data source in the available state, perform a synchronization operation of synchronizing the second data to the second data source, thereby enabling bidirectional synchronization of data and meeting increasingly complex data synchronization requirements.

Based on the principle of bidirectional synchronization of data, in the case that the first data source and the second data source provide data storage services for different users of the same service, the following situations exist in this embodiment:

1. After the first data source is available, in the process of synchronizing the first data to be synchronized and the second data to be synchronized, the data synchronization middleware acquires third data to be synchronized generated by the second data source in an available state, and performs synchronization operation of synchronizing the third data to be synchronized to the first data source;

2. After the first data to be synchronized is completed, when the first data source and the second data source are both available, the data synchronization middleware continuously performs a synchronization operation of synchronizing the second data to be synchronized to the second data source, and continuously performs a synchronization operation of synchronizing the third data to be synchronized to the first data source;

3. before the first data source is unavailable, data needing to be synchronized are generated in the first data source and the second data source, and the data synchronization middleware performs synchronization operation of synchronizing the data needing to be synchronized in the first data source to the second data source and synchronization operation of synchronizing the data needing to be synchronized in the second data source to the first data source.

Through the situations, the data in different data sources can be synchronized and consistent, the data are mutually backed up, and therefore the effect of data integrity is guaranteed, for example, the data of Beijing payment users and the data of Shanghai payment users are stored in the first data source and the second data source, so that the data are mutually backed up, and the data integrity is guaranteed.

Fig. 2 is a schematic flow chart of a data synchronization method according to an embodiment of the present disclosure, where the method is applied to the data synchronization middleware in fig. 1, and the data synchronization middleware is executed by the data synchronization middleware, and as described above, the data synchronization middleware is configured to synchronize data between a first data source and a second data source, where the second data source is a backup data source during a period when the first data source is unavailable, and as shown in fig. 2, the flow includes the following steps:

Step S202, obtaining first data to be synchronized generated by a second data source in a first data source unavailable period, and executing a synchronization operation of synchronizing the first data to be synchronized to the first data source after the first data source is changed to an available state;

in step S204, during the process of synchronizing the first data to be synchronized, the second data to be synchronized generated by the first data source in the available state is obtained, and a synchronization operation of synchronizing the second data to be synchronized to the second data source is performed.

During periods when the first data source is not available, the second data source is a backup data source for the first data source. In one case, the second data source and the first data source correspond to different users of the same service, respectively provide data storage services for different users of the same service, and during the unavailable period of the first data source, the second data source serves as a backup data source of the first data source to provide data storage services for both the user corresponding to the first data source and the user corresponding to the second data source. In another case, the first data source is a cold backup data source of the second data source, when the first data source is in an available state, the first data source is preferentially used for data storage, and when the first data source is unavailable, the second data source is used for data storage.

In the step S202, the data synchronization middleware obtains first data to be synchronized generated by the second data source during the period that the first data source is unavailable, and specifically includes:

(a1) Receiving a data change log based on a message form, wherein the data change log is transmitted by a second data source during the unavailable period of the first data source;

(a2) Acquiring a data identifier of first data to be synchronized generated by a second data source in a first data source unavailable period from a data change log;

(a3) And sending a data acquisition request based on the message form to a second data source according to the data identification of the first data to be synchronized so as to acquire the first data to be synchronized.

In act (a 1), the data synchronization middleware receives a message-based data change log sent by the second data source during a period when the first data source is unavailable. In this embodiment, the second data source may send a data change log to the data synchronization middleware at regular time in an available state, where the data change log records a data identifier of changed data, and the data change log may be sent based on a message, for example, based on http or TCP message, and is carried in the message, so that the data synchronization middleware receives the data change log based on the message form, which is sent by the second data source during the period when the first data source is unavailable. The data changes within the second data source may be exemplified by: the second data source is a backup data source serving as the first data source, provides data storage service for the Shanghai payment user corresponding to the second data source, and also provides data storage service for the Beijing payment user, and if the Shanghai payment user or the Beijing payment user executes payment related operations, such as payment operations, data in the second data source is changed, and a data change log records the data identification of the changed data. The data that is changed may be a file.

In the act (a 2), the data synchronization middleware obtains, from the data change log, a data identifier of first data to be synchronized generated by the second data source during a period when the first data source is unavailable, where the act specifically includes:

(a21) Reading a data identifier of data changed by the second data source in the unavailable period of the first data source from the data change log;

(a22) And selecting the data identification of the first data to be synchronized from the data identifications of the changed data according to a preset data to be synchronized determining rule.

First, a data identifier of data of which the second data source recorded in the data change log is changed during a period in which the first data source is not available is read. Then, according to a preconfigured data determining rule such as preconfigured data to be synchronized by a developer, determining a data identifier of the data to be synchronized, searching the data identifier of the data to be synchronized in the data identifier of the changed data, if so, determining the searched data identifier as a data identifier of the first data to be synchronized, and if not, determining that the data to be changed does not exist in the data recorded in the data change log, so that the data identifier of the first data to be synchronized does not exist. The data identification of the data to be synchronized is recorded in the data to be synchronized determining rule, and the rule can be configured by a developer according to the need and stored in the data synchronization middleware.

In the act (a 3), after the data identifier of the first data to be synchronized is obtained, a data obtaining request based on a message form is sent to the second data source, where the request may be an http or TCP request, and the request carries the data identifier of the first data to be synchronized, so as to obtain the first data to be synchronized.

In step S202, the data synchronization middleware performs a synchronization operation of synchronizing the first data to be synchronized to the first data source after determining that the first data source is changed to the available state. The data synchronization middleware may periodically send a heartbeat communication signal to the first data source, so as to monitor whether the first data source is in an available state, and after determining that the first data source is in the available state, perform a synchronization operation of synchronizing the first data to be synchronized to the first data source.

In one embodiment, performing a synchronization operation to synchronize first data to be synchronized to a first data source includes:

(b1) Determining a second volume address of the first data to be synchronized in the first data source according to a preset mapping relation between the volume address in the first data source and the volume address in the second data source and the first volume address of the first data to be synchronized in the second data source;

(b2) Determining a second storage path of the first data to be synchronized under the second volume address according to the first storage path of the first data to be synchronized under the first volume address;

(b3) And sending a data writing message to the first data source according to a second storage path of the first data to be synchronized under the second volume address so as to write the first data to be synchronized into the first data source.

In act (b 1), the data synchronization middleware pre-stores a pre-configured mapping relationship between volume addresses in the first data source and volume addresses in the second data source, such as configured by a developer. When the first data source and the second data source are both server clusters, a plurality of servers in the first data source may form a volume, the first data source may include one or more volumes, a plurality of servers in the second data source may form a volume, the second data source may include one or more volumes, and a mapping relationship exists between the volume in the first data source and the volume in the second data source, for example, a volume A1 in the first data source corresponds to a volume B1 in the second data source, and a volume A2 in the first data source corresponds to a volume B2 in the second data source. In this step, according to the mapping relationship between the volume address in the first data source and the volume address in the second data source, which are preconfigured by the developer, and the first volume address, such as A1, of the first data to be synchronized in the second data source, the second volume address, such as B1, of the first data to be synchronized in the first data source is determined.

In the act (b 2), a second storage path of the first data to be synchronized under the second volume address is determined according to the first storage path of the first data to be synchronized under the first volume address, and the first storage path of the first data to be synchronized under the first volume address may be the same as the second storage path of the first data to be synchronized under the second volume address, that is, the same data storage paths are set in the first data source and the second data source, so that data synchronization between the first data source and the second data source is facilitated. For example, the first storage path of the first data to be synchronized under the first volume address is A1/2021/3, which indicates that the first data to be synchronized is under the first data source and the folder under the name 2021 under the volume A1 is under the name 3, and then the second storage path of the first data to be synchronized under the second volume address is determined to be B1/2021/3, which indicates that the first data to be synchronized is under the second data source and the folder under the name 2021 under the volume B1 is under the name 3.

In act (b 3), a data write message is sent to the first data source to write the first data to be synchronized to the first data source according to a second storage path of the first data to be synchronized at the second volume address. For example, as in the example above, the first data to be synchronized is written under the folder under the name 2021 in the second data source. In the action, according to a second storage path of the first data to be synchronized under the second volume address, a data writing message is sent to the first data source, which specifically includes:

(b31) If the data with the same data identification as the first data to be synchronized is stored in the second storage path, a data coverage instruction is generated, and a first data writing sub-message is sent to the first data source; the first data writing sub-message carries first data to be synchronized, a second storage path of the first data to be synchronized under a second volume address and a data covering instruction, and the data covering instruction is used for indicating to cover the data of the same data identifier by the first data to be synchronized;

(b32) If the data with the same data identification as the first data to be synchronized is not stored in the second storage path, generating a data adding instruction and sending a second data writing sub-message to the first data source; the second data writing sub-message carries first data to be synchronized, a second storage path of the first data to be synchronized under a second volume address, and a data increasing instruction, wherein the data increasing instruction is used for indicating the first data to be synchronized to be increased under the second storage path.

Considering that when the data with the same data identifier as the first data to be synchronized is stored in the second storage path, the data with the same data identifier may be the data stored by the previous user, the timeliness is weaker than that of the first data to be synchronized, so that whether the data with the same data identifier as the first data to be synchronized is stored in the second storage path is checked, if so, a data coverage instruction is generated, and the first data writing sub-message is sent to the first data source, so that the data with the same data identifier is covered by the first data to be synchronized. If the first data is not stored, a data adding instruction is generated, and the second data writing sub-message is sent to the first data source, so that the first data to be synchronized is added under the second storage path.

Through the step S202, after the first data source is changed from unavailable to available, the first data to be synchronized generated in the second data source during the first data source unavailable period can be synchronized to the first data source, so that the consistency of the data in the first data source and the second data source is maintained.

In one scenario, the first data source and the second data source provide data storage services for different users of the same service, and after the first data source is unavailable, the second data source provides data storage services for the users corresponding to the first data source, and the second data source stores changed data generated by the users corresponding to the first data source in the period that the first data source is unavailable. The developer can configure the above data to be synchronized determining rule, and the identifier of the data to be synchronized in the configuring rule can be a predetermined data identifier of important user data, so that after the first data source is available, the important data is obtained from the changed data generated by the user corresponding to the first data source in the period that the first data source is unavailable, and the important data is synchronized from the second data source to the first data source. In this scenario, the data synchronization middleware may synchronize all of the changed data generated by the user corresponding to the first data source during the period in which the first data source is unavailable.

In another scenario, the first data source is a cold backup data source, and after the first data source is unavailable, the second data source provides a data storage service for a user corresponding to the first data source, and the second data source stores changed data generated by the user corresponding to the first data source in a period in which the first data source is unavailable. The developer can configure the above data to be synchronized determining rule, and the identifier of the data to be synchronized in the configuring rule can be a predetermined data identifier of important user data, so that after the first data source is available, the important data is obtained from the changed data generated by the user corresponding to the first data source in the period that the first data source is unavailable, and the important data is synchronized from the second data source to the first data source. In this scenario, the data synchronization middleware may synchronize all of the changed data generated by the user corresponding to the first data source during the period in which the first data source is unavailable.

In the step S204, during the process of synchronizing the first data to be synchronized, the second data to be synchronized generated by the first data source in the available state is obtained, and the synchronization operation of synchronizing the second data to be synchronized to the second data source is performed.

After the first data source is restored to the available state, data change may be performed based on the data read-write operation of the user, and part or all of the changed data is the second data to be synchronized generated by the first data source in the available state. In consideration of the consistency of the data in the first data source and the second data source, in this step, during the process of synchronizing the first data source to be synchronized, the second data to be synchronized generated by the first data source in the available state is obtained, and the synchronization operation of synchronizing the second data to be synchronized to the second data source is performed, so that the consistency of the data in the first data source and the second data source is ensured.

In this step, obtaining second data to be synchronized generated by the first data source in the available state includes:

(c1) Receiving a data change log based on a message form, which is sent by a first data source in an available state;

(c2) Acquiring a data identifier of second data to be synchronized generated by the first data source in an available state from the data change log;

(c3) And sending a data acquisition request based on the message form to the first data source according to the data identification of the second data to be synchronized so as to acquire the second data to be synchronized.

The specific procedure of the actions (c 1) to (c 3) may be referred to the previous description of the actions (a 1) to (a 3), and will not be repeated here.

In this step, a synchronization operation for synchronizing the second data to be synchronized to the second data source is performed, including:

(d1) Determining a fourth volume address of the second data to be synchronized in the second data source according to a preset mapping relation between the volume address in the first data source and the volume address in the second data source and a third volume address of the second data to be synchronized in the first data source;

(d2) Determining a fourth storage path of the second data to be synchronized under a fourth volume address according to a third storage path of the second data to be synchronized under the third volume address;

(d3) And sending a data writing message to the second data source according to a fourth storage path of the second data to be synchronized under the fourth volume address so as to write the second data to be synchronized into the second data source.

The specific procedure of the actions (d 1) to (d 3) may be referred to the previous description of the actions (b 1) to (b 3), and will not be repeated here.

In one scenario, the first data source and the second data source provide data storage services for different users of the same service, and after the first data source is unavailable, the second data source provides data storage services for the users corresponding to the first data source, and the second data source stores changed data generated by the users corresponding to the first data source in the period that the first data source is unavailable. The developer can configure the above data to be synchronized determining rule, and the identifier of the data to be synchronized in the configuring rule can be a predetermined data identifier of important user data, so that after the first data source is available, the important data is obtained from the changed data generated by the user corresponding to the first data source in the period that the first data source is unavailable, and the important data is synchronized from the second data source to the first data source. In this scenario, the data synchronization middleware may synchronize all of the changed data generated by the user corresponding to the first data source during the period in which the first data source is unavailable. In this scenario, after the first data source is restored to be available, the data changed in the first data source may be synchronized to the second data source in whole or in part based on the configuration of the user in step S204.

In another scenario, the first data source is a cold backup data source, and after the first data source is unavailable, the second data source provides a data storage service for a user corresponding to the first data source, and the second data source stores changed data generated by the user corresponding to the first data source in a period in which the first data source is unavailable. The developer can configure the above data to be synchronized determining rule, and the identifier of the data to be synchronized in the configuring rule can be a predetermined data identifier of important user data, so that after the first data source is available, the important data is obtained from the changed data generated by the user corresponding to the first data source in the period that the first data source is unavailable, and the important data is synchronized from the second data source to the first data source. In this scenario, the data synchronization middleware may synchronize all of the changed data generated by the user corresponding to the first data source during the period in which the first data source is unavailable. In this scenario, after the first data source is restored to be available, the data changed in the first data source may be synchronized to the second data source in whole or in part based on the configuration of the user in step S204.

Therefore, through the method in the embodiment, the bidirectional synchronization between the first data source and the second data source can be realized through the data synchronization middleware, the data synchronization middleware is communicated with the first data source and the second data source based on the form of information, and the data to be synchronized is acquired and written, so that the data synchronization process is simple and convenient, any improvement on the first data source and the second data source is not needed, and the working process of the first data source and the second data source is not influenced by the data synchronization.

In one scenario, the first data source and the second data source provide data storage services for different users of the same service, and therefore, in this embodiment, in the process of synchronizing the first data to be synchronized and the second data to be synchronized, third data to be synchronized generated by the second data source in an available state is acquired, and a synchronization operation of synchronizing the third data to be synchronized to the first data source is performed, because the data in the second data source may also be changed based on user operation transmission and synchronization to the first data source is also necessary. The specific process of this action may refer to the foregoing description, for example, the identifier of the third data to be synchronized is obtained from the data change log, the third data to be synchronized is obtained from the second data source based on the identifier, and the synchronization operation of synchronizing the third data to be synchronized to the first data source is performed through the predetermined mapping relationship between the volume addresses, which may not be described any more. By the action, when the data in the second data source is changed, the data in the second data source can be synchronized to the first data source, so that the data consistency in the first data source and the second data source is ensured through the bidirectional synchronization of the data.

In this scenario, after the synchronization of the first data to be synchronized is completed, the synchronization operation of synchronizing the second data to be synchronized to the second data source is continuously performed, and the synchronization operation of synchronizing the third data to be synchronized to the first data source is continuously performed, that is, in a state where both the first data source and the second data source are available, the data in the first data source is continuously synchronized to the second data source, and the data in the second data source is continuously synchronized to the first data source, so that the data consistency in the first data source and the second data source is ensured through the bidirectional synchronization of the data.

In another scenario, the second data source is a cold backup data source of the first data source, and the second data source does not provide a service for the user, after the synchronization of the first data to be synchronized is completed, the data in the second data source does not need to be synchronized to the first data source, but instead, the first data source provides a service for the user, and the synchronization operation of synchronizing the second data to be synchronized in the first data source to the second data source is continuously performed.

Fig. 3a is a schematic diagram of data synchronization according to an embodiment of the present disclosure, as shown in fig. 3a, where a first data source and a second data source provide data storage services for different users of the same service, and after the first data source is restored to be available from an unavailable state, the second data source synchronizes first data to be synchronized to the first data source, and at the same time, the first data source synchronizes second data to be synchronized to the second data source according to the foregoing description.

Fig. 3b is a schematic diagram of data synchronization according to another embodiment of the present disclosure, as shown in fig. 3b, where the first data source and the second data source provide data storage services for different users of the same service, and according to the foregoing description, based on fig. 3a, in a process of synchronizing the first data to be synchronized and the second data to be synchronized, the second data source synchronizes the third data to be synchronized to the first data source.

Fig. 3c is a schematic diagram of data synchronization according to another embodiment of the present disclosure, as shown in fig. 3c, where the first data source and the second data source provide data storage services for different users of the same service, and according to the foregoing description, after the first data to be synchronized is completed, the second data source continuously synchronizes the third data to be synchronized to the first data source, and the first data source continuously synchronizes the second data to be synchronized to the second data source on the basis of fig. 3 b.

Fig. 4a is a schematic diagram of data synchronization according to another embodiment of the present disclosure, as shown in fig. 4a, where the second data source is a cold backup data source of the first data source, and after the first data source is restored to be available from the unavailable state, the second data source synchronizes the first data to be synchronized with the first data source, and at the same time, the first data source synchronizes the second data to be synchronized with the second data source according to the foregoing description.

Fig. 4b is a schematic diagram of data synchronization according to another embodiment of the present disclosure, as shown in fig. 4b, where the second data source is a cold backup data source of the first data source, and according to the foregoing description, based on fig. 4a, after the synchronization of the first data to be synchronized is completed, synchronization operation of synchronizing the second data to be synchronized to the second data source is continuously performed.

In summary, the data synchronization method in this embodiment has at least the following technical effects:

(1) The data synchronization is realized through an independent data synchronization middleware, the data synchronization process does not influence the work of a data source, and the failure of the data synchronization middleware does not influence the normal operation of the data source;

(2) When the first data source is available, the data in the first data source can be synchronized to the second data source, when the first data source is recovered from unavailable to available, the data in the unavailable period of the first data source can be synchronized to the first data source, and meanwhile, the data in the available state of the first data source is synchronized to the second data source, so that the integrity of the data in the unavailable period of the first data source is ensured, and the bidirectional synchronization of the data is realized.

The embodiment of the present disclosure also provides a data synchronization device, based on the same concept, for executing the data synchronization method provided in the embodiment. Fig. 5 is a schematic block diagram of a data synchronization device according to an embodiment of the present disclosure, where the device is applied to a data synchronization middleware, and the data synchronization middleware is configured to synchronize data between a first data source and a second data source, and the second data source is a backup data source during a period when the first data source is unavailable, as shown in fig. 5, where the device includes:

A first synchronization module 51, configured to acquire first to-be-synchronized data generated by the second data source during the unavailable period, and perform a synchronization operation of synchronizing the first to-be-synchronized data to the first data source after the first data source is changed to an available state;

And the second synchronization module 52 is configured to acquire second data to be synchronized generated by the first data source in an available state during the process of synchronizing the first data to be synchronized, and perform a synchronization operation of synchronizing the second data to be synchronized to the second data source.

Optionally, the first data source and the second data source provide data storage services for different users of the same service; the apparatus further comprises: and the third synchronization module is used for acquiring third data to be synchronized generated by the second data source in an available state in the process of synchronizing the first data to be synchronized and the second data to be synchronized, and executing the synchronization operation of synchronizing the third data to be synchronized to the first data source.

Optionally, the first synchronization module is specifically configured to: receiving a message-based data change log transmitted by the second data source during the unavailable period; acquiring a data identifier of first data to be synchronized generated by the second data source in the unavailable period from the data change log; and sending a data acquisition request based on a message form to the second data source according to the data identification of the first data to be synchronized so as to acquire the first data to be synchronized.

Optionally, the first synchronization module is further specifically configured to: reading a data identifier of the data of which the second data source is changed in the unavailable period from the data change log; and selecting the data identification of the first data to be synchronized from the data identifications of the changed data according to a preset data to be synchronized determining rule.

Optionally, the first synchronization module is specifically configured to: determining a second volume address of the first data to be synchronized in the first data source according to a preset mapping relation between the volume address in the first data source and the volume address in the second data source and the first volume address of the first data to be synchronized in the second data source; determining a second storage path of the first data to be synchronized under the second volume address according to a first storage path of the first data to be synchronized under the first volume address; and sending a data writing message to the first data source according to a second storage path of the first data to be synchronized under the second volume address so as to write the first data to be synchronized into the first data source.

Optionally, the first synchronization module is further specifically configured to: if the data with the same data identification as the first data to be synchronized is stored in the second storage path, a data coverage instruction is generated, and a first data writing sub-message is sent to the first data source; the first data writing sub-message carries the first data to be synchronized, a second storage path of the first data to be synchronized under the second volume address and the data covering instruction, and the data covering instruction is used for indicating to cover the data of the same data identifier by using the first data to be synchronized; if the data with the same data identification as the first data to be synchronized is not stored in the second storage path, a data adding instruction is generated, and a second data writing sub-message is sent to the first data source; the second data writing sub-message carries the first data to be synchronized, a second storage path of the first data to be synchronized under the second volume address, and the data increasing instruction, where the data increasing instruction is used to instruct to increase the first data to be synchronized under the second storage path.

Optionally, the second synchronization module is specifically configured to: receiving a data change log based on a message form, which is sent by the first data source in an available state; acquiring a data identifier of second data to be synchronized generated by the first data source in an available state from the data change log; and sending a data acquisition request based on a message form to the first data source according to the data identification of the second data to be synchronized so as to acquire the second data to be synchronized.

Optionally, the second synchronization module is specifically configured to: determining a fourth volume address of the second data to be synchronized in the second data source according to a preset mapping relation between the volume address in the first data source and the volume address in the second data source and a third volume address of the second data to be synchronized in the first data source; determining a fourth storage path of the second data to be synchronized under the fourth volume address according to a third storage path of the second data to be synchronized under the third volume address; and sending a data writing message to the second data source according to a fourth storage path of the second data to be synchronized under the fourth volume address so as to write the second data to be synchronized into the second data source.

Optionally, the apparatus further comprises: and the first repeated synchronization module is used for continuously executing the synchronization operation of synchronizing the second data to be synchronized to the second data source and continuously executing the synchronization operation of synchronizing the third data to be synchronized to the first data source after the synchronization of the first data to be synchronized is completed.

Optionally, the second data source is a cold backup data source of the first data source; the apparatus further comprises: and the second repeated synchronization module is used for continuously executing the synchronization operation of synchronizing the second data to be synchronized to the second data source after the synchronization of the first data to be synchronized is completed.

The data synchronization device provided in the embodiment of the present specification has at least the following beneficial effects:

And in the process of synchronizing the first data to be synchronized, acquiring second data to be synchronized generated by the first data source in an available state, and executing the synchronization operation of synchronizing the second data to be synchronized to the second data source, thereby realizing bidirectional synchronization of the data and meeting increasingly complex data synchronization requirements.

The data synchronization device provided in the embodiment shown in fig. 5 can implement all the method steps in the above-mentioned data synchronization method embodiment, and achieve the same functions and effects, which are not described herein again.

The embodiment of the present disclosure also provides a data synchronization device, based on the same concept, for executing the data synchronization method provided in the embodiment. Fig. 6 is a schematic structural diagram of a data synchronization device according to an embodiment of the present disclosure, as shown in fig. 6, where the data synchronization device may have a relatively large difference due to different configurations or performances, and may include one or more processors 1001 and a memory 1002, where the memory 1002 may store one or more storage applications or data. Wherein the memory 1002 may be transient storage or persistent storage. The application programs stored in the memory 1002 may include one or more modules (not shown), each of which may include a series of computer-executable instruction information in the data synchronization apparatus. Still further, the processor 1001 may be configured to communicate with the memory 1002 and execute a series of computer executable instruction information in the memory 1002 on a data synchronization device. The data synchronization device may also include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input/output interfaces 1005, one or more keyboards 1006, and the like.

In one case, the data synchronization device is the data synchronization middleware, and in another case, the data synchronization middleware is implemented by a plurality of servers, and the data synchronization device is any one server in the data synchronization middleware. In a particular embodiment, a data synchronization device includes a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to implement the following:

The data synchronization device provided in the embodiment of the present specification has at least the following advantages:

The data synchronization device provided in the embodiment shown in fig. 6 may implement all the method steps of the above-mentioned data synchronization method embodiment, and achieve the same functions and effects, which are not described herein again.

The embodiment of the present disclosure also provides a storage medium for executing the data synchronization method provided in the above embodiment, based on the same concept. In a specific embodiment, the storage medium may be a usb disk, an optical disk, a hard disk, or the like, where the storage medium is applied to a data synchronization middleware for synchronizing data between a first data source and a second data source, where the second data source is a backup data source during a period when the first data source is not available, and where the storage medium is used to store computer executable instructions that when executed by a processor implement the following procedures:

The storage medium provided in the embodiments of the present disclosure stores computer executable instruction information that, when executed by a processor, has at least the following advantages:

The computer executable instruction information stored in the storage medium provided in the embodiments of the present disclosure may implement all the method steps of the embodiments of the data synchronization method when executed by the processor, and achieve the same functions and effects, which are not described herein.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the 90s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable GATE ARRAY, FPGA)) is an integrated circuit whose logic functions are determined by user programming of the device. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented with "logic compiler (logic compiler)" software, which is similar to the software compiler used in program development and writing, and the original code before being compiled is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but HDL is not just one, but a plurality of kinds, such as ABEL(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description Language)、Confluence、CUPL(Cornell University Programming Language)、HDCal、JHDL(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby Hardware Description Language), and VHDL (Very-High-SPEED INTEGRATED Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application SPECIFIC INTEGRATED Circuits (ASICs), programmable logic controllers, and embedded microcontrollers, examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instruction information. These computer program instruction information may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instruction information, which is executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instruction information may also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instruction information stored in the computer readable memory produce an article of manufacture including instruction information means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instruction information may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instruction information which is executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instruction information, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

The application may be described in the general context of computer-executable instruction information, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. A data synchronization method, applied to a data synchronization middleware for synchronizing data between a first data source and a second data source, the second data source being a backup data source during a period when the first data source is unavailable, the method comprising:

2. The method of claim 1, wherein the first data source and the second data source provide data storage services for different users of the same service; the method further comprises the steps of:

And in the process of synchronizing the first data to be synchronized and the second data to be synchronized, acquiring third data to be synchronized generated by the second data source in an available state, and executing the synchronization operation of synchronizing the third data to be synchronized to the first data source.

3. The method of claim 1, wherein obtaining the first data to be synchronized generated by the second data source during the unavailable period comprises:

Receiving a message-based data change log transmitted by the second data source during the unavailable period;

Acquiring a data identifier of first data to be synchronized generated by the second data source in the unavailable period from the data change log;

And sending a data acquisition request based on a message form to the second data source according to the data identification of the first data to be synchronized so as to acquire the first data to be synchronized.

4. A method according to claim 3, wherein obtaining, from the data change log, a data identification of first data to be synchronized generated by the second data source during the unavailable period, comprises:

Reading a data identifier of the data of which the second data source is changed in the unavailable period from the data change log;

and selecting the data identification of the first data to be synchronized from the data identifications of the changed data according to a preset data to be synchronized determining rule.

5. The method of claim 1, wherein performing a synchronization operation that synchronizes the first data to be synchronized to the first data source comprises:

Determining a second volume address of the first data to be synchronized in the first data source according to a preset mapping relation between the volume address in the first data source and the volume address in the second data source and the first volume address of the first data to be synchronized in the second data source;

determining a second storage path of the first data to be synchronized under the second volume address according to a first storage path of the first data to be synchronized under the first volume address;

and sending a data writing message to the first data source according to a second storage path of the first data to be synchronized under the second volume address so as to write the first data to be synchronized into the first data source.

6. The method of claim 5, wherein sending a data write message to the first data source according to a second storage path of the first data to be synchronized at the second volume address comprises:

If the data with the same data identification as the first data to be synchronized is stored in the second storage path, a data coverage instruction is generated, and a first data writing sub-message is sent to the first data source; the first data writing sub-message carries the first data to be synchronized, a second storage path of the first data to be synchronized under the second volume address and the data covering instruction, and the data covering instruction is used for indicating to cover the data of the same data identifier by using the first data to be synchronized;

If the data with the same data identification as the first data to be synchronized is not stored in the second storage path, a data adding instruction is generated, and a second data writing sub-message is sent to the first data source; the second data writing sub-message carries the first data to be synchronized, a second storage path of the first data to be synchronized under the second volume address, and the data increasing instruction, where the data increasing instruction is used to instruct to increase the first data to be synchronized under the second storage path.

7. The method of claim 1, wherein performing a synchronization operation that synchronizes the second data to be synchronized to the second data source comprises:

Determining a fourth volume address of the second data to be synchronized in the second data source according to a preset mapping relation between the volume address in the first data source and the volume address in the second data source and a third volume address of the second data to be synchronized in the first data source;

determining a fourth storage path of the second data to be synchronized under the fourth volume address according to a third storage path of the second data to be synchronized under the third volume address;

And sending a data writing message to the second data source according to a fourth storage path of the second data to be synchronized under the fourth volume address so as to write the second data to be synchronized into the second data source.

8. A data synchronization device for use in a data synchronization middleware for synchronizing data between a first data source and a second data source, the second data source being a backup data source during a period when the first data source is unavailable, the device comprising:

9. A data synchronization device for synchronizing data between a first data source and a second data source, the second data source being a backup data source during periods when the first data source is unavailable, the device comprising:

A processor; and

10. A storage medium for use in data synchronization middleware for synchronizing data between a first data source and a second data source, the second data source being a backup data source during periods when the first data source is unavailable, the storage medium for storing computer executable instructions that when executed by a processor implement the following: