CN112860796B - Method, apparatus, device and storage medium for synchronizing data - Google Patents
Method, apparatus, device and storage medium for synchronizing data Download PDFInfo
- Publication number
- CN112860796B CN112860796B CN202110179205.0A CN202110179205A CN112860796B CN 112860796 B CN112860796 B CN 112860796B CN 202110179205 A CN202110179205 A CN 202110179205A CN 112860796 B CN112860796 B CN 112860796B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- metadata
- cluster
- response
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
Abstract
The application discloses a method, a device, equipment and a storage medium for synchronizing data, and relates to the technical field of cloud computing and cloud storage. The specific implementation scheme is as follows: in response to receiving a data synchronization request sent by a source data cluster, analyzing the data synchronization request, and determining target data to be synchronized and target metadata of the target data; reading local metadata of target data stored in the data cluster; determining whether a preset synchronization condition is met according to the target metadata and the local metadata; and synchronizing the target data and the target metadata in the data cluster in response to determining that the preset synchronization condition is met. The implementation mode can realize the data consistency of each data cluster.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to the field of cloud computing and cloud storage technologies, and in particular, to a method, an apparatus, a device, and a storage medium for synchronizing data.
Background
The object storage cluster generally stores data in a multi-copy or erasure code mode, so that the reliability of the data in the same cluster is ensured, and the data loss caused by single disk or single machine faults is avoided. Meanwhile, in order to meet the low delay of data communication in the object storage cluster, all machines of the set of object storage clusters are generally in the same machine room. In the scene that the requirement on data security is higher, not only data is required to be stored in one machine room, but also two machine rooms in the same city are required, and the problem that one set of machine room cannot provide online service due to abnormality such as power failure is avoided. Or two machine rooms in different places, so that data loss caused by natural disasters such as earthquake in one city is avoided.
The data of the objects can be stored simultaneously among multiple machine rooms, which requires the clusters to communicate with each other, and the data is synchronized to the object storage clusters on another machine room in a copying mode. In order to ensure low delay of online service, the industry generally adopts an asynchronous replication mode to synchronize data into another cluster, namely, after the data of the object is stored in the cluster, the result of successful processing is immediately returned to the client. The background then asynchronously copies the data into other clusters.
Meanwhile, in order to meet the requirement that the client can access data nearby, the transmission delay is reduced, and multiple clusters are required to provide services for upper-layer services at the same time, namely, the client can access any cluster nearby, and the multi-active cluster is hereinafter referred to as a multi-active cluster. The data uploaded in one cluster is guaranteed to be synchronized to other clusters. This presents a challenge for data consistency, i.e. uploading or deleting the same object no matter which set of clusters the client accesses, and finally the object information on these clusters is guaranteed to be completely consistent.
Disclosure of Invention
A method, apparatus, device, and storage medium for synchronizing data are provided.
According to a first aspect, there is provided a method for synchronizing data, comprising: in response to receiving a data synchronization request sent by a source data cluster, analyzing the data synchronization request, and determining target data to be synchronized and target metadata of the target data; reading local metadata of target data stored in the data cluster; determining whether a preset synchronization condition is met according to the target metadata and the local metadata; and synchronizing the target data and the target metadata in the data cluster in response to determining that the preset synchronization condition is met.
According to a second aspect, there is provided an apparatus for synchronizing data, comprising: the request receiving unit is configured to respond to the received data synchronization request sent by the source data cluster, analyze the data synchronization request and determine target data to be synchronized and target metadata of the target data; a metadata reading unit configured to read local metadata of target data stored in the present data cluster; a condition judgment unit configured to determine whether a preset synchronization condition is satisfied according to the target metadata and the local metadata; and a data synchronization unit configured to synchronize the target data and the target metadata in the present data cluster in response to determining that the preset synchronization condition is satisfied.
According to a third aspect, there is provided an electronic device for processing data, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.
According to a fourth aspect, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method as described in the first aspect.
According to a fifth aspect, a computer program product comprising a computer program which, when executed by a processor, implements the method as described in the first aspect.
According to the technology of the application, a data processing method is provided, asynchronous storage can be realized in a distributed storage system, and data consistency is maintained.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for better understanding of the present solution and do not constitute a limitation of the present application. Wherein:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;
FIG. 2 is a flow chart of one embodiment of a method for synchronizing data according to the present application;
FIG. 3 is a schematic illustration of one application scenario of a method for synchronizing data according to the present application;
FIG. 4 is a flow chart of another embodiment of a method for synchronizing data according to the present application;
FIG. 5 is a schematic structural diagram of one embodiment of an apparatus for synchronizing data according to the present application;
fig. 6 is a block diagram of an electronic device for implementing a method for synchronizing data of an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the methods for synchronizing data or the apparatus for synchronizing data of the present application may be applied.
As shown in fig. 1, a system architecture 100 may include a terminal device 101, a network 102, a server 103, and a distributed data cluster 104. Network 102 is the medium used to provide communication links between terminal device 101 and server 103, and distributed data cluster 104. Network 102 may include various connection types such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 103 via the network 102 using the terminal device 101 to enable access to the distributed data cluster 104 for writing or deleting data or the like. The terminal device 101 may have various communication client applications installed thereon, such as a database access class application or the like.
The terminal device 101 may be hardware or software. When the terminal device 101 is hardware, it may be a variety of electronic devices including, but not limited to, smartphones, tablet computers, electronic book readers, car-mounted computers, laptop and desktop computers, and the like. When the terminal apparatus 101 is software, it can be installed in the above-listed electronic apparatus. Which may be implemented as multiple software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein.
The server 103 may be a server providing various services, such as a database server providing support for the distributed data cluster 104. The database server may write or delete data to the distributed data cluster 104 in response to a request of the terminal device 101, and feed back a processing result of the data to the terminal device 101.
The server 103 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster formed by a plurality of servers, or as a single server. When the server 103 is software, it may be implemented as a plurality of software or software modules (for example, to provide distributed services), or may be implemented as a single software or software module. The present invention is not particularly limited herein.
It should be noted that the method for synchronizing data provided in the embodiments of the present application is generally performed by the distributed data cluster 104. Accordingly, means for synchronizing data is typically provided in the distributed data cluster 104.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of one embodiment of a method for synchronizing data according to the present application is shown. The method for synchronizing data of the present embodiment includes the steps of:
in step 201, in response to receiving the data synchronization request sent by the source data cluster, the data synchronization request is parsed, and target data to be synchronized and metadata of the target data are determined.
Target data identification, target modification time, target operation type, and target operation identification.
In this embodiment, an executing body of the method for synchronizing data (e.g., any one of the distributed data clusters 104 shown in fig. 1) may receive a data synchronization request sent by another data cluster. Other data clusters may be referred to herein as source data clusters, and execution bodies may be referred to as target data clusters. The source data cluster may process the data after receiving a data processing request sent by the user terminal. And after the processing is completed, sending a data synchronization request to the target data cluster.
The destination data cluster may analyze the data synchronization request to determine the destination data to be synchronized and metadata for the destination data. Here, the target data may be various data to be synchronized, such as data of a document, multimedia, or the like. The metadata of the target data may include a target data identification, a target modification time, a target operation type, and a target operation identification. The target data identification may be an identification of the target data, such as a title, a storage path, and the like. The target modification time may be a data update time of the target data, and may include a write time and a delete time. The target operation types may include write operations and delete operations. It is understood that modifications to the data may be considered a combination of write operations and/or delete operations. The target operation identifier may be used to uniquely represent the target operation described above, which has global uniqueness. It should be noted that, in this embodiment, each data cluster may interact with a user terminal, and may receive a processing request of a user for data, so that data may be processed.
Step 202, reading local metadata of the target data stored in the data cluster.
After the target data cluster analyzes the data synchronization request, the target data cluster can be searched according to the data identification in the target metadata. Here, metadata of target data stored in the present data cluster may be referred to as local metadata. After the local metadata is found, the local metadata may be read. It is understood that the contents included in the local metadata may be identical to the contents in the target metadata. Including, for example, modification time, data identification, operation type, etc. To distinguish content in the local metadata from content in the target metadata, each item in the local metadata may be referred to as a local modification time, a local data identification, a local operation type, and the like.
Step 203, determining whether the preset synchronization condition is satisfied according to the target metadata and the local metadata.
The target data cluster can judge whether the preset synchronization condition is met according to the target metadata and the local metadata. Specifically, the preset synchronization condition may include that the target data identifier is consistent with the local data identifier, or that the target modification time is consistent with the local modification time, or that the duration between the target modification time and the local modification time is not more than a preset threshold, or the like.
In step 204, in response to determining that the preset synchronization condition is satisfied, the target data and the target metadata are synchronized in the present data cluster.
In this embodiment, after determining that the preset synchronization condition is satisfied, the data cluster synchronizes the target data and the target metadata. Specifically, the data cluster may synchronize target data first, and synchronize metadata after synchronization of the target data is completed. When synchronizing the target data, the data cluster can update the data stored in the data cluster according to the target operation type under the condition that the target operation identification is consistent with the local identification. In synchronizing the target metadata, the local metadata may be directly replaced with the target metadata. Thus, the synchronization of data among the data clusters can be realized.
With continued reference to fig. 3, a schematic diagram of one application scenario of a method for synchronizing data according to the present application is shown. In the application scenario of fig. 3, the source data cluster 301 sends a data synchronization request to the destination data cluster 302, and the destination data cluster 302 analyzes the data synchronization request to obtain the target data a and the target metadata b1 of the target data. At the same time, the destination data cluster 302 reads the local metadata b2 of the target data a stored in the present data cluster. By comparing the target metadata b1 with the local metadata b2, it is determined that the target modification time in the target metadata b1 is later than the local modification time in the local metadata b2, and the preset synchronization condition is determined to be satisfied. At this time, the destination data cluster 302 may synchronize the above-described destination data a and destination metadata b1.
The method for synchronizing data provided by the embodiment of the application can synchronize data in different data clusters, and ensure the consistency of data among the data clusters.
With continued reference to fig. 4, a flow 400 of another embodiment of a method for synchronizing data according to the present application is shown. In this embodiment, the metadata may include a data identifier, a modification time, an operation type, an operation identifier, and a deletion identifier. Wherein, the content in the target metadata may be referred to as a target data identifier, a target modification time, a target operation type, a target operation identifier, and a target deletion identifier. The content in the local metadata may be referred to as a local data identifier, a local modification time, a local operation type, a local operation identifier, and a local deletion identifier. The method of the present embodiment may include the steps of:
in step 401, in response to receiving the data synchronization request sent by the source data cluster, the data synchronization request is parsed, and target data to be synchronized and metadata of the target data are determined.
Step 402, reading local metadata of the target data stored in the data cluster.
In some optional implementations of this embodiment, the destination data cluster may first determine whether local metadata of the target data is stored in the data cluster, and if the local metadata of the target data is stored in the data cluster, the local metadata may be read to determine modification time and operation identifier in the target data. If the data cluster does not store the metadata of the target data, the data cluster is indicated that the target data is not stored.
In step 403, in response to determining that the local modification time is earlier than the target modification time, it is determined that the preset synchronization condition is satisfied.
It will be appreciated that if the target modification time is earlier than the local modification time, which means that the target data is processed by the data cluster after the target data in the source data cluster is modified, in this case, the target data cannot be directly processed, the preset synchronization condition is not satisfied. If the local modification time is earlier than the target modification time, which means that the processing of the target data by the data cluster is performed before the processing of the target data by the source data cluster, in this case, the target data can be processed, and the preset synchronization condition is satisfied.
In step 404, in response to determining that the preset synchronization condition is satisfied, the target data and the target metadata are synchronized in the present data cluster.
Specifically, the step 404 may specifically include the following steps:
step 4041, in response to determining that the target operation type is a deletion operation, setting a deletion identifier of the local metadata to a preset value; the local metadata is updated with the data based on the target data identification, the target operation type, the target operation identification, and the target modification time.
If the target operation type is a delete operation, the destination data cluster may set a delete flag of the local metadata stored in the present data cluster to a preset value. Here, the preset value indicates that the target data is to be deleted, and the non-preset value indicates that the target data does not need to be deleted. The executing body may then replace the content in the local metadata with the content in the target metadata.
Step 4042, in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster; in response to receiving target data from the source data cluster, the target data is synchronized to the present data cluster.
If the target operation type is a write operation, the destination data cluster may send a data pull request to the source data cluster. The data pull request is for requesting target data from a source data cluster. The data pull request may include a target data identification and a target operation identification. If target data is received from a source data cluster, the target data may be synchronized to the present data cluster.
In some alternative implementations of the present embodiment, the step 4042 may be implemented by including the following steps, not shown in fig. 4: receiving an operation identifier of target data from a source data cluster; in response to determining that the received operation identifier is the same as the target operation identifier, synchronizing the target data to the current data cluster.
In this implementation manner, the destination data cluster may also receive an operation identifier for the target data sent by the source data cluster. Then, whether the received operation identifier is the same as the target operation identifier is judged. If so, the target data may be synchronized to the current data cluster. In this way, a reconfirmation of the data can be achieved,
in response to determining that the synchronization of the target data is complete, a synchronization success message is sent to the source data cluster, step 405.
If the target data is synchronized in the data cluster, the target data cluster can send a synchronization success message to the source data cluster to inform the source data cluster that the synchronization of the target data is completed.
Step 406, in response to determining that the preset deletion condition is satisfied, determining whether the target operation identifier is consistent with the operation identifier in the metadata of the target data stored in the present data cluster; in response to determining that the two are consistent, the target data is deleted.
In this embodiment, the destination data cluster or some electronic device in communication with the destination data cluster may monitor whether the preset deletion condition is satisfied in real time. If so, it may be further determined whether the target operation identifier is consistent with the operation identifier in the metadata of the target data stored in the present data cluster. If the target data is consistent, indicating that there is no error, the target data may be deleted. In some specific applications, step 406 may be performed by a recycler. The recycler may delete data identified as the preset value from the metadata periodically or quantitatively. Before deleting, whether the target operation identifier in the recoverer is consistent with the operation identifier in the metadata of the target data stored in the current data cluster or not can be judged, so that the target data can be reconfirmed, and the accuracy of data deleting is ensured.
The method for synchronizing data provided by the embodiment of the application can firstly delete the data when deleting the data and then delete the data when the deleting condition is met, thereby realizing the delayed deletion of the data and further ensuring the consistency of the data.
With further reference to fig. 5, as an implementation of the method shown in the foregoing figures, the present application provides an embodiment of an apparatus for synchronizing data, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus is specifically applicable to various electronic devices.
As shown in fig. 5, the data processing apparatus 500 of the present embodiment includes: a request receiving unit 501, a metadata reading unit 502, a condition judging unit 503, and a data synchronizing unit 504.
The request receiving unit 501 is configured to, in response to receiving a data synchronization request sent by the source data cluster, parse the data synchronization request and determine target data to be synchronized and target metadata of the target data.
The metadata reading unit 502 is configured to read local metadata of target data stored in the present data cluster.
The condition judgment unit 503 is configured to determine whether a preset synchronization condition is satisfied according to the target metadata and the local metadata.
The data synchronization unit 504 is configured to synchronize the target data and the target metadata in the present data cluster in response to determining that the preset synchronization condition is satisfied.
In some alternative implementations of the present embodiment, the target metadata includes a target modification time and the local metadata includes a local modification time. The condition judgment unit 503 may be further configured to: in response to determining that the local modification time is earlier than the target modification time, it is determined that the preset synchronization condition is satisfied.
In some alternative implementations of the present embodiment, the target metadata includes a target operation type and the local metadata includes a delete identifier. The data synchronization unit 504 may be further configured to: setting a deletion identifier of the local metadata to a preset value in response to determining that the target operation type is a deletion operation; the local metadata is updated with the data based on the target data identification, the target operation type, the target operation identification, and the target modification time.
In some alternative implementations of the present embodiment, the data synchronization unit 504 may be further configured to: in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster, the data pull request including a target data identifier and a target operation identifier; in response to receiving target data from the source data cluster, the target data is synchronized to the present data cluster.
In some alternative implementations of the present embodiment, the data synchronization unit 504 may be further configured to: receiving an operation identifier of target data from a source data cluster; in response to determining that the received operation identifier is the same as the target operation identifier, synchronizing the target data to the current data cluster.
In some optional implementations of the present embodiment, the apparatus 500 may further include a message sending unit, not shown in fig. 5, configured to: and in response to determining that the synchronization of the target data is completed, sending a synchronization success message to the source data cluster.
In some optional implementations of the present embodiment, the apparatus 500 may further include a data deletion unit, not shown in fig. 5, configured to: determining whether the target operation identifier is consistent with the operation identifier in the metadata of the target data stored in the data cluster or not in response to determining that the preset deletion condition is met; in response to determining that the two are consistent, the target data is deleted.
It should be understood that the units 501 to 504 described in the apparatus 500 for synchronizing data correspond to the respective steps in the method described with reference to fig. 2. Thus, the operations and features described above with respect to the method for synchronizing data are equally applicable to the apparatus 500 and the units contained therein, and are not described in detail herein.
According to embodiments of the present application, there is also provided an electronic device, a readable storage medium and a computer program product.
Fig. 6 shows a block diagram of an electronic device 600 performing a method for synchronizing data according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.
As shown in fig. 6, the device 600 includes a processor 601 which may perform various suitable actions and processes in accordance with a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a memory 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 may also be stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An I/O interface (input/output interface) 605 is also connected to the bus 604.
Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; memory 608, e.g., magnetic disk, optical disk, etc.; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 601 performs the various methods and processes described above, such as methods for synchronizing data. For example, in some embodiments, the method for synchronizing data may be implemented as a computer software program tangibly embodied on a machine-readable storage medium, such as memory 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When a computer program is loaded into RAM 603 and executed by processor 601, one or more of the steps of the method for synchronizing data described above may be performed. Alternatively, in other embodiments, processor 601 may be configured to perform the method for synchronizing data in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present application may be written in any combination of one or more programming languages. The program code described above may be packaged into a computer program product. These program code or computer program products may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor 601, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable storage medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable storage medium may be a machine-readable signal storage medium or a machine-readable storage medium. The machine-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solutions of the present application are achieved, and the present application is not limited herein.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (10)
1. A method for synchronizing data, comprising:
responding to a received data synchronization request sent by a source data cluster, analyzing the data synchronization request, and determining target data to be synchronized and target metadata of the target data;
reading local metadata of the target data stored in the data cluster;
determining whether a preset synchronization condition is met according to the target metadata and the local metadata;
synchronizing the target data and the target metadata at the present data cluster in response to determining that the preset synchronization condition is satisfied; the target metadata includes a target modification time, and the local metadata includes a local modification time; and
the determining whether the preset synchronization condition is met according to the target metadata and the local metadata comprises the following steps:
determining that the preset synchronization condition is met in response to determining that the local modification time is earlier than the target modification time;
the target metadata comprises a target operation type, and the local metadata comprises a deletion identifier; and synchronizing the target data and the target metadata at the local data cluster, including:
in response to determining that the target operation type is a deletion operation, setting a deletion identifier of the local metadata to a preset value, wherein the preset value represents that the target data is to be deleted;
updating the local metadata with the target data identifier, the target operation type, the target operation identifier and the target modification time;
determining whether the target operation identifier is consistent with the operation identifier in the metadata of the target data stored in the data cluster or not in response to the fact that the preset deleting condition is met;
and deleting the target data in response to determining that the target data and the target data are consistent.
2. The method of claim 1, wherein the synchronizing the target data and the target metadata at the present data cluster comprises:
in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster, the data pull request including a target data identifier and a target operation identifier;
in response to receiving target data from the source data cluster, the target data is synchronized to the present data cluster.
3. The method of claim 2, wherein the synchronizing the target data to the present data cluster comprises:
receiving an operation identification of the target data from the source data cluster;
and synchronizing the target data to the data cluster in response to determining that the received operation identifier is the same as the target operation identifier.
4. The method of claim 1, wherein the method further comprises:
and sending a synchronization success message to the source data cluster in response to determining that the synchronization of the target data is completed.
5. An apparatus for synchronizing data, comprising:
a request receiving unit configured to parse the data synchronization request in response to receiving the data synchronization request sent by the source data cluster, and determine target data to be synchronized and target metadata of the target data;
a metadata reading unit configured to read local metadata of the target data stored in the present data cluster;
a condition judgment unit configured to determine whether a preset synchronization condition is satisfied according to the target metadata and the local metadata;
a data synchronization unit configured to synchronize the target data and the target metadata at the present data cluster in response to determining that the preset synchronization condition is satisfied;
the target metadata includes a target modification time, and the local metadata includes a local modification time; and
the condition judgment unit is further configured to:
determining that the preset synchronization condition is met in response to determining that the local modification time is earlier than the target modification time; the target metadata comprises a target operation type, and the local metadata comprises a deletion identifier; and
the data synchronization unit is further configured to:
in response to determining that the target operation type is a deletion operation, setting a deletion identifier of the local metadata to a preset value, wherein the preset value represents that the target data is to be deleted;
updating the local metadata with the target data identifier, the target operation type, the target operation identifier and the target modification time;
a data deleting unit configured to: determining whether the target operation identifier is consistent with the operation identifier in the metadata of the target data stored in the data cluster or not in response to the fact that the preset deleting condition is met; and deleting the target data in response to determining that the target data and the target data are consistent.
6. The apparatus of claim 5, wherein the data synchronization unit is further configured to:
in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster, the data pull request including a target data identifier and a target operation identifier;
in response to receiving target data from the source data cluster, the target data is synchronized to the present data cluster.
7. The apparatus of claim 6, wherein the data synchronization unit is further configured to:
receiving an operation identification of the target data from the source data cluster;
and synchronizing the target data to the data cluster in response to determining that the received operation identifier is the same as the target operation identifier.
8. The apparatus of claim 5, wherein the apparatus further comprises a messaging unit configured to:
and sending a synchronization success message to the source data cluster in response to determining that the synchronization of the target data is completed.
9. An electronic device that performs a method for synchronizing data, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.
10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110179205.0A CN112860796B (en) | 2021-02-08 | 2021-02-08 | Method, apparatus, device and storage medium for synchronizing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110179205.0A CN112860796B (en) | 2021-02-08 | 2021-02-08 | Method, apparatus, device and storage medium for synchronizing data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860796A CN112860796A (en) | 2021-05-28 |
| CN112860796B true CN112860796B (en) | 2023-07-25 |
Family
ID=75989483
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110179205.0A Active CN112860796B (en) | 2021-02-08 | 2021-02-08 | Method, apparatus, device and storage medium for synchronizing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860796B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115730016B (en) * | 2023-01-28 | 2023-09-19 | 北京国科天迅科技有限公司 | Data synchronization method, system, device, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9015123B1 (en) * | 2013-01-16 | 2015-04-21 | Netapp, Inc. | Methods and systems for identifying changed data in an expandable storage volume |
| CN107491343A (en) * | 2017-09-08 | 2017-12-19 | 中国电子科技集团公司第二十八研究所 | A kind of across cluster resource scheduling system based on cloud computing |
| CN108769212A (en) * | 2018-05-31 | 2018-11-06 | 康键信息技术（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111752960A (en) * | 2020-06-28 | 2020-10-09 | 北京百度网讯科技有限公司 | Data processing method and device |
| CN112114757A (en) * | 2020-09-28 | 2020-12-22 | 北京百度网讯科技有限公司 | Storage method and system in object storage system, computing device and medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10496676B2 (en) * | 2014-08-29 | 2019-12-03 | Netapp Inc. | Synchronization cache seeding |
-
2021
- 2021-02-08 CN CN202110179205.0A patent/CN112860796B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9015123B1 (en) * | 2013-01-16 | 2015-04-21 | Netapp, Inc. | Methods and systems for identifying changed data in an expandable storage volume |
| CN107491343A (en) * | 2017-09-08 | 2017-12-19 | 中国电子科技集团公司第二十八研究所 | A kind of across cluster resource scheduling system based on cloud computing |
| CN108769212A (en) * | 2018-05-31 | 2018-11-06 | 康键信息技术（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111752960A (en) * | 2020-06-28 | 2020-10-09 | 北京百度网讯科技有限公司 | Data processing method and device |
| CN112114757A (en) * | 2020-09-28 | 2020-12-22 | 北京百度网讯科技有限公司 | Storage method and system in object storage system, computing device and medium |
Non-Patent Citations (3)
| Title |
|---|
| Kun Qian ; Zhongzhi Luan ; Zuowei Zhang ; Hailong Yang ; Yaqi Yang ; Depei Qian.Data Analysis and Synchronization on Inter-Continent Data Placement Laboratory.IEEE.2016,全文. * |
| 云存储系统架构与关键技术研究;李明娟;邸海涛;张鹏;任晓瑞;;信息通信(第03期);全文 * |
| 空间数据存储对象的元数据可伸缩性管理;李锐;林艳萍;徐正全;冯蔚;;计算机应用研究(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860796A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10489378B2 (en) | Detection and resolution of conflicts in data synchronization | |
| US20200004464A1 (en) | Method and apparatus for storing data | |
| CN111338834B (en) | Data storage method and device | |
| CN112860796B (en) | Method, apparatus, device and storage medium for synchronizing data | |
| CN113067860B (en) | Method, apparatus, device, medium and product for synchronizing information | |
| CN111444148B (en) | Data transmission method and device based on MapReduce | |
| CN112613964A (en) | Account checking method, account checking device, account checking equipment and storage medium | |
| CN113326038B (en) | Method, apparatus, device, storage medium and program product for providing service | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN115640280A (en) | Data migration method and device | |
| CN113641688B (en) | Node updating method, related device and computer program product | |
| CN112887426B (en) | Information stream pushing method and device, electronic equipment and storage medium | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN114706774A (en) | Interface test method, device, equipment and storage medium | |
| CN113342759A (en) | Content sharing method, device, equipment and storage medium | |
| CN113553216A (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN113569144B (en) | Method, device, equipment, storage medium and program product for searching promotion content | |
| US20230132173A1 (en) | Data reading method, device and storage medium | |
| CN114745438B (en) | Cache data processing method, device and system of multi-data center and electronic equipment | |
| CN113609145B (en) | Database processing method, device, electronic equipment, storage medium and product | |
| CN113313196B (en) | Labeling data processing method, related device and computer program product | |
| CN114500438B (en) | File sharing method and device, electronic equipment and storage medium | |
| CN112749042B (en) | Application running method and device | |
| CN115510036A (en) | Data migration method, device, equipment and storage medium | |
| CN113282571A (en) | Data transfer method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |