CN112860796A - Method, apparatus, device and storage medium for synchronizing data - Google Patents
Method, apparatus, device and storage medium for synchronizing data Download PDFInfo
- Publication number
- CN112860796A CN112860796A CN202110179205.0A CN202110179205A CN112860796A CN 112860796 A CN112860796 A CN 112860796A CN 202110179205 A CN202110179205 A CN 202110179205A CN 112860796 A CN112860796 A CN 112860796A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- metadata
- cluster
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device, equipment and a storage medium for synchronizing data, and relates to the technical field of cloud computing and cloud storage. The specific implementation scheme is as follows: in response to receiving a data synchronization request sent by a source data cluster, analyzing the data synchronization request, and determining target data to be synchronized and target metadata of the target data; reading local metadata of target data stored in the data cluster; determining whether a preset synchronization condition is met according to the target metadata and the local metadata; and in response to determining that the preset synchronization condition is met, synchronizing the target data and the target metadata in the data cluster. The implementation mode can realize the data consistency of each data cluster.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to the field of cloud computing and cloud storage technologies, and in particular, to a method, an apparatus, a device, and a storage medium for synchronizing data.
Background
The object storage cluster generally stores data in a multi-copy or erasure code mode, so that the reliability of the data in the same cluster is ensured, and the data loss caused by the failure of a single disk or a single machine is avoided. Meanwhile, in order to meet the requirement of low time delay of data communication in the object storage cluster, all machines of one set of object storage cluster are generally in the same machine room. In a scene with higher requirements for data safety, data are required to be stored in one machine room, two machine rooms in the same city are also required, and the problem that one set of machine rooms cannot provide online service due to power failure and other abnormalities is avoided. Or two machine rooms in different places, so that data loss caused by natural disasters such as earthquake and the like in one city is avoided.
The data of the objects can be stored simultaneously among a plurality of computer rooms, which requires that clusters communicate with each other and synchronize the data to the object storage cluster on another computer room in a replication mode. In general, in order to ensure low delay of online service, data is synchronized to another cluster by asynchronous replication, that is, after the data of the object is stored in the cluster, a result of successful processing is returned to the client immediately. The background then asynchronously copies the data to other clusters.
Meanwhile, in order to meet the requirement that a client can access data nearby and reduce transmission delay, multiple clusters are required to provide service for upper-layer services at the same time, that is, the client can access any cluster nearby, which is hereinafter referred to as a multi-active cluster. Data uploaded in a certain cluster is guaranteed to be synchronized to other clusters. This poses a challenge to data consistency, that is, no matter which cluster a client accesses, the same object is uploaded or deleted, and finally the object information on the clusters is guaranteed to be completely consistent.
Disclosure of Invention
A method, apparatus, device, and storage medium for synchronizing data are provided.
According to a first aspect, there is provided a method for synchronizing data, comprising: in response to receiving a data synchronization request sent by a source data cluster, analyzing the data synchronization request, and determining target data to be synchronized and target metadata of the target data; reading local metadata of target data stored in the data cluster; determining whether a preset synchronization condition is met according to the target metadata and the local metadata; and in response to determining that the preset synchronization condition is met, synchronizing the target data and the target metadata in the data cluster.
According to a second aspect, there is provided an apparatus for synchronizing data, comprising: the request receiving unit is configured to respond to the received data synchronization request sent by the source data cluster, analyze the data synchronization request, and determine target data to be synchronized and target metadata of the target data; a metadata reading unit configured to read local metadata of target data stored in the present data cluster; a condition judging unit configured to determine whether a preset synchronization condition is satisfied according to the target metadata and the local metadata; and the data synchronization unit is configured to synchronize the target data and the target metadata in the data cluster in response to determining that the preset synchronization condition is met.
According to a third aspect, there is provided an electronic device for processing data, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.
According to a fourth aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method as described in the first aspect.
According to a fifth aspect, a computer program product comprising a computer program which, when executed by a processor, implements the method as described in the first aspect.
The technology according to the application provides a data processing method which can realize asynchronous storage in a distributed storage system and keep the consistency of data.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present application may be applied;
FIG. 2 is a flow diagram for one embodiment of a method for synchronizing data according to the present application;
FIG. 3 is a schematic diagram of an application scenario of a method for synchronizing data according to the present application;
FIG. 4 is a flow diagram of another embodiment of a method for synchronizing data according to the present application;
FIG. 5 is a schematic block diagram illustrating one embodiment of an apparatus for synchronizing data according to the present application;
FIG. 6 is a block diagram of an electronic device for implementing a method for synchronizing data according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the present method for synchronizing data or apparatus for synchronizing data may be applied.
As shown in fig. 1, system architecture 100 may include end devices 101, a network 102, servers 103, and distributed data clusters 104. Network 102 is used to provide a medium for communication links between terminal devices 101 and servers 103, and distributed data clusters 104. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use a terminal device 101 to interact with a server 103 over a network 102 to gain access to a distributed data cluster 104 to write or delete data, etc. The terminal device 101 may have various communication client applications installed thereon, such as a database access application.
The terminal apparatus 101 may be hardware or software. When the terminal device 101 is hardware, it may be various electronic devices including, but not limited to, a smart phone, a tablet computer, an e-book reader, a car computer, a laptop portable computer, a desktop computer, and the like. When the terminal apparatus 101 is software, it can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The server 103 may be a server that provides various services, such as a database server that provides support for the distributed data cluster 104. The database server may write or delete data into the distributed data cluster 104 in response to a request from the terminal apparatus 101, and feed back a processing result of the data to the terminal apparatus 101.
The server 103 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the server 103 is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be noted that the method for synchronizing data provided by the embodiments of the present application is generally performed by the distributed data cluster 104. Accordingly, the means for synchronizing data is generally disposed in the distributed data cluster 104.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of a method for synchronizing data in accordance with the present application is shown. The method for synchronizing data of the embodiment comprises the following steps:
Target data identification, target modification time, target operation type, and target operation identification.
In this embodiment, an execution subject of the method for synchronizing data (for example, any one of the distributed data clusters 104 shown in fig. 1) may receive a data synchronization request sent by another data cluster. The other data clusters may be referred to herein as source data clusters and the execution entity may be referred to as a target data cluster. The source data cluster may process the data after receiving a data processing request sent by the user terminal. And after the processing is finished, sending a data synchronization request to the target data cluster.
The destination data cluster may analyze the data synchronization request to determine target data to be synchronized and metadata of the target data. Here, the target data may be various data to be synchronized, such as document, multimedia, and the like. The metadata of the target data may include a target data identification, a target modification time, a target operation type, and a target operation identification. The target data identification may be an identification of the target data, such as a title, a storage path, and so forth. The target modification time may be a data update time of the target data, and may include a write time and a delete time. The target operation type may include a write operation and a delete operation. It will be appreciated that the modification of the data may be considered a combination of write operations and/or delete operations. The target operation identifier can be used for uniquely representing the target operation, and has global uniqueness. It should be noted that, in this embodiment, each data cluster may interact with a user terminal, and may receive a request for processing data from a user, so that the data may be processed.
After the target data cluster has analyzed the data synchronization request, the target data cluster can be searched according to the data identifier in the target metadata. Here, metadata of target data stored by the present data cluster may be referred to as local metadata. After the local metadata is found, the local metadata can be read. It is to be understood that the items of content included in the local metadata may be the same as the items of content in the target metadata. For example, all include modification time, data identification, operation type, and the like. To distinguish content in the local metadata from content in the target metadata, items in the local metadata may be referred to as local modification time, local data identification, local operation type, and so on.
The target data cluster can judge whether the preset synchronization condition is met according to the target metadata and the local metadata. Specifically, the preset synchronization condition may include that the target data identifier is consistent with the local data identifier, or that the target modification time is consistent with the local modification time, or that a time duration between the target modification time and the local modification time does not exceed a preset threshold, and the like.
In this embodiment, after the data cluster determines that the preset synchronization condition is satisfied, target data and target metadata are synchronized in the data cluster. Specifically, the data cluster may synchronize the target data first, and synchronize the metadata after completing synchronization of the target data. When the target data is synchronized, the data cluster can update the data stored in the data cluster according to the target operation type under the condition that the target operation identifier is determined to be consistent with the local identifier again. In synchronizing the target metadata, the local metadata may be directly replaced with the target metadata. In this way, synchronization of data between data clusters can be achieved.
With continued reference to fig. 3, a schematic diagram of one application scenario of a method for synchronizing data according to the present application is shown. In the application scenario of fig. 3, the source data cluster 301 sends a data synchronization request to the destination data cluster 302, and the destination data cluster 302 parses the data synchronization request to obtain the target data a and the target metadata b1 of the target data. Meanwhile, the destination data cluster 302 reads the local metadata b2 of the target data a stored in the present data cluster. By comparing the target metadata b1 and the local metadata b2, it is determined that the target modification time in the target metadata b1 is later than the local modification time in the local metadata b2, and it is assumed that the preset synchronization condition is satisfied. At this point, destination data cluster 302 may synchronize target data a and target metadata b1 as described above.
The method for synchronizing data provided by the above embodiments of the present application can synchronize data in different data clusters, and ensure consistency of data among the data clusters.
With continued reference to FIG. 4, a flow 400 of another embodiment of a method for synchronizing data according to the present application is shown. In this embodiment, the metadata may include a data identifier, a modification time, an operation type, an operation identifier, and a deletion identifier. The content in the target metadata may be referred to as a target data identifier, a target modification time, a target operation type, a target operation identifier, and a target deletion identifier. The content in the local metadata may be referred to as a local data identification, a local modification time, a local operation type, a local operation identification, and a local deletion identification. The method of the embodiment may include the steps of:
In some optional implementation manners of this embodiment, the destination data cluster may first determine whether the local metadata of the target data is stored in the data cluster, and if the local metadata of the target data is stored in the data cluster, the local metadata may be read, and the modification time and the operation identifier in the target data are determined. If the data cluster does not store the metadata of the target data, the data cluster does not store the target data.
In response to determining that the local modification time is earlier than the target modification time, it is determined that a preset synchronization condition is satisfied, step 403.
It can be understood that, if the target modification time is earlier than the local modification time, which indicates that the data cluster has processed the target data after the target data in the source data cluster is modified, in this case, the target data cannot be directly processed, and the preset synchronization condition is not satisfied. If the local modification time is earlier than the target modification time, which indicates that the target data can be processed in the case that the target data is processed before the source data cluster, the preset synchronization condition is satisfied.
Specifically, the step 404 may specifically include the following steps:
If the target operation type is a delete operation, the destination data cluster may set a delete flag of the local metadata stored in the present data cluster to a preset value. Here, the preset value indicates that the target data is to be deleted, and the non-preset value indicates that the target data does not need to be deleted. Then, the executing body may replace each item of content in the local metadata with each item of content in the target metadata.
If the target operation type is a write operation, the destination data cluster may send a data pull request to the source data cluster. The data pull request is used to request target data from the source data cluster. The data pull request may include a target data identification and a target operation identification. If target data from the source data cluster is received, the target data may be synchronized to the present data cluster.
In some optional implementations of this embodiment, step 4042 may be implemented by including the following steps not shown in fig. 4: receiving operation identification of target data from a source data cluster; in response to determining that the received operation identification is the same as the target operation identification, the target data is synchronized to the present data cluster.
In this implementation, the destination data cluster may also receive an operation identifier for the target data sent by the source data cluster. And then judging whether the received operation identifier is the same as the target operation identifier. If so, the target data may be synchronized to the present data cluster. In this way, a further confirmation of the data can be achieved,
in response to determining that the synchronization of the target data is complete, step 405 sends a synchronization success message to the source data cluster.
If the target data is synchronized in the data cluster, the target data cluster may send a synchronization success message to the source data cluster to inform the source data cluster that the synchronization of the target data is completed.
In this embodiment, the destination data cluster or an electronic device communicating with the destination data cluster may monitor whether the preset deletion condition is met in real time. If yes, whether the target operation identifier is consistent with the operation identifier in the metadata of the target data stored in the data cluster can be further judged. If consistent, indicating no error, the target data may be deleted. In some specific applications, step 406 may be performed by a retriever. The recoverer may delete the data identified as the preset value in the metadata periodically or quantitatively. Before deletion, whether the target operation identifier in the recoverer is consistent with the operation identifier in the metadata of the target data stored in the current data cluster or not can be achieved, so that reconfirmation of the target data can be achieved, and the accuracy of data deletion is guaranteed.
The method for synchronizing data provided by the above embodiment of the present application can delete data first when deleting data, and delete data when a deletion condition is satisfied, thereby implementing delayed deletion of data, and ensuring consistency of data.
With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, the present application provides an embodiment of an apparatus for synchronizing data, which corresponds to the method embodiment shown in fig. 2, and which is particularly applicable to various electronic devices.
As shown in fig. 5, the apparatus 500 for processing data of the present embodiment includes: a request receiving unit 501, a metadata reading unit 502, a condition judging unit 503, and a data synchronizing unit 504.
The request receiving unit 501 is configured to, in response to receiving a data synchronization request sent by a source data cluster, parse the data synchronization request, and determine target data to be synchronized and target metadata of the target data.
A metadata reading unit 502 configured to read local metadata of target data stored in the present data cluster.
A condition judging unit 503 configured to determine whether a preset synchronization condition is satisfied according to the target metadata and the local metadata.
A data synchronization unit 504 configured to synchronize the target data and the target metadata in the present data cluster in response to determining that the preset synchronization condition is satisfied.
In some optional implementations of this embodiment, the target metadata includes a target modification time and the local metadata includes a local storage time. The condition determining unit 503 may be further configured to: in response to determining that the local modification time is earlier than the target modification time, determining that a preset synchronization condition is satisfied.
In some optional implementations of this embodiment, the target metadata includes a target operation type, and the local metadata includes a deletion identification. The data synchronization unit 504 may be further configured to: in response to determining that the target operation type is a deletion operation, setting a deletion identifier of the local metadata to a preset value; and updating the metadata according to the target data identifier, the target operation type, the target operation identifier and the target modification time.
In some optional implementations of this embodiment, the data synchronization unit 504 may be further configured to: in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster, the data pull request including a target data identifier and a target operation identifier; in response to receiving the target data from the source data cluster, the target data is synchronized to the local data cluster.
In some optional implementations of this embodiment, the data synchronization unit 504 may be further configured to: receiving operation identification of target data from a source data cluster; in response to determining that the received operation identification is the same as the target operation identification, the target data is synchronized to the present data cluster.
In some optional implementations of this embodiment, the apparatus 500 may further include a message sending unit, not shown in fig. 5, configured to: in response to determining that the synchronization of the target data is complete, a synchronization success message is sent to the source data cluster.
In some optional implementations of this embodiment, the apparatus 500 may further include a data deleting unit, not shown in fig. 5, configured to: in response to determining that the preset deletion condition is met, determining whether the target operation identifier is consistent with an operation identifier in metadata of the target data stored in the data cluster; in response to determining that the two are consistent, the target data is deleted.
It should be understood that the units 501 to 504, which are described in the apparatus 500 for synchronizing data, correspond to the respective steps in the method described with reference to fig. 2, respectively. Thus, the operations and features described above for the method for synchronizing data are equally applicable to the apparatus 500 and the units included therein and will not be described again here.
The application also provides an electronic device, a readable storage medium and a computer program product according to the embodiment of the application.
Fig. 6 shows a block diagram of an electronic device 600 that performs a method for synchronizing data according to an embodiment of the application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 6, the device 600 includes a processor 601 that may perform various appropriate actions and processes in accordance with a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a memory 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I/O interface (input/output interface) 605 is also connected to the bus 604.
A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a memory 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
Processor 601 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of processor 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 601 performs the various methods and processes described above, such as the method for synchronizing data. For example, in some embodiments, the method for synchronizing data may be implemented as a computer software program tangibly embodied in a machine-readable storage medium, such as memory 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the processor 601, one or more steps of the method for synchronizing data described above may be performed. Alternatively, in other embodiments, the processor 601 may be configured by any other suitable means (e.g., by way of firmware) to perform the method for synchronizing data.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. The program code described above may be packaged as a computer program product. These program code or computer program products may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor 601, causes the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable storage medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable storage medium may be a machine-readable signal storage medium or a machine-readable storage medium. A machine-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server can be a cloud Server, also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service ("Virtual Private Server", or simply "VPS").
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution of the present application can be achieved, and the present invention is not limited thereto.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (17)
1. A method for synchronizing data, comprising:
responding to a received data synchronization request sent by a source data cluster, analyzing the data synchronization request, and determining target data to be synchronized and target metadata of the target data;
reading local metadata of the target data stored in the data cluster;
determining whether a preset synchronization condition is met according to the target metadata and the local metadata;
and in response to determining that the preset synchronization condition is met, synchronizing the target data and the target metadata in the data cluster.
2. The method of claim 1, wherein the target metadata comprises a target modification time, the local metadata comprises a local storage time; and
the determining whether a preset synchronization condition is satisfied according to the target metadata and the local metadata includes:
determining that the preset synchronization condition is satisfied in response to determining that the local modification time is earlier than the target modification time.
3. The method of claim 1, wherein the target metadata comprises a target operation type, the local metadata comprises a deletion identification; and
the synchronizing the target data and the target metadata in the present data cluster includes:
in response to determining that the target operation type is a delete operation, setting a delete identifier of the local metadata to a preset value;
and updating the metadata by utilizing the data according to the target data identifier, the target operation type, the target operation identifier and the target modification time.
4. The method of claim 1, wherein the synchronizing the target data and the target metadata at the present data cluster comprises:
in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster, where the data pull request includes a target data identifier and a target operation identifier;
in response to receiving target data from the source data cluster, synchronizing the target data to a local data cluster.
5. The method of claim 4, wherein the synchronizing the target data to the local data cluster comprises:
receiving an operation identification of the target data from the source data cluster;
in response to determining that the received operation identification is the same as the target operation identification, synchronizing the target data to a local data cluster.
6. The method of claim 1, wherein the method further comprises:
in response to determining that synchronization of the target data is complete, sending a synchronization success message to the source data cluster.
7. The method of claim 4, wherein the method further comprises:
in response to determining that a preset deletion condition is met, determining whether the target operation identifier is consistent with an operation identifier in metadata of the target data stored in the data cluster;
deleting the target data in response to determining that the two are consistent.
8. An apparatus for synchronizing data, comprising:
the data synchronization device comprises a request receiving unit, a data synchronization processing unit and a data synchronization processing unit, wherein the request receiving unit is configured to respond to a data synchronization request sent by a source data cluster, analyze the data synchronization request, and determine target data to be synchronized and target metadata of the target data;
a metadata reading unit configured to read local metadata of the target data stored in a present data cluster;
a condition judging unit configured to determine whether a preset synchronization condition is satisfied according to the target metadata and the local metadata;
a data synchronization unit configured to synchronize the target data and the target metadata at a present data cluster in response to determining that the preset synchronization condition is satisfied.
9. The apparatus of claim 8, wherein the target metadata comprises a target modification time, the local metadata comprises a local storage time; and
the condition judging unit is further configured to:
determining that the preset synchronization condition is satisfied in response to determining that the local modification time is earlier than the target modification time.
10. The apparatus of claim 8, wherein the target metadata comprises a target operation type, the local metadata comprises a deletion identification; and
the data synchronization unit is further configured to:
in response to determining that the target operation type is a delete operation, setting a delete identifier of the local metadata to a preset value;
and updating the metadata by utilizing the data according to the target data identifier, the target operation type, the target operation identifier and the target modification time.
11. The apparatus of claim 8, wherein the data synchronization unit is further configured to:
in response to determining that the target operation type is a write operation, sending a data pull request to the source data cluster, where the data pull request includes a target data identifier and a target operation identifier;
in response to receiving target data from the source data cluster, synchronizing the target data to a local data cluster.
12. The apparatus of claim 11, wherein the data synchronization unit is further configured to:
receiving an operation identification of the target data from the source data cluster;
in response to determining that the received operation identification is the same as the target operation identification, synchronizing the target data to a local data cluster.
13. The apparatus of claim 8, wherein the apparatus further comprises a message sending unit configured to:
in response to determining that synchronization of the target data is complete, sending a synchronization success message to the source data cluster.
14. The apparatus of claim 11, wherein the apparatus further comprises a data deletion unit configured to:
in response to determining that a preset deletion condition is met, determining whether the target operation identifier is consistent with an operation identifier in metadata of the target data stored in the data cluster;
deleting the target data in response to determining that the two are consistent.
15. An electronic device that performs a method for synchronizing data, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.
17. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110179205.0A CN112860796B (en) | 2021-02-08 | 2021-02-08 | Method, apparatus, device and storage medium for synchronizing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110179205.0A CN112860796B (en) | 2021-02-08 | 2021-02-08 | Method, apparatus, device and storage medium for synchronizing data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860796A true CN112860796A (en) | 2021-05-28 |
| CN112860796B CN112860796B (en) | 2023-07-25 |
Family
ID=75989483
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110179205.0A Active CN112860796B (en) | 2021-02-08 | 2021-02-08 | Method, apparatus, device and storage medium for synchronizing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860796B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115730016A (en) * | 2023-01-28 | 2023-03-03 | 北京国科天迅科技有限公司 | Data synchronization method, system, device, computer equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9015123B1 (en) * | 2013-01-16 | 2015-04-21 | Netapp, Inc. | Methods and systems for identifying changed data in an expandable storage volume |
| US20160063083A1 (en) * | 2014-08-29 | 2016-03-03 | Netapp, Inc. | Synchronization cache seeding |
| CN107491343A (en) * | 2017-09-08 | 2017-12-19 | 中国电子科技集团公司第二十八研究所 | A kind of across cluster resource scheduling system based on cloud computing |
| CN108769212A (en) * | 2018-05-31 | 2018-11-06 | 康键信息技术（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111752960A (en) * | 2020-06-28 | 2020-10-09 | 北京百度网讯科技有限公司 | Data processing method and device |
| CN112114757A (en) * | 2020-09-28 | 2020-12-22 | 北京百度网讯科技有限公司 | Storage method and system in object storage system, computing device and medium |
-
2021
- 2021-02-08 CN CN202110179205.0A patent/CN112860796B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9015123B1 (en) * | 2013-01-16 | 2015-04-21 | Netapp, Inc. | Methods and systems for identifying changed data in an expandable storage volume |
| US20160063083A1 (en) * | 2014-08-29 | 2016-03-03 | Netapp, Inc. | Synchronization cache seeding |
| CN107491343A (en) * | 2017-09-08 | 2017-12-19 | 中国电子科技集团公司第二十八研究所 | A kind of across cluster resource scheduling system based on cloud computing |
| CN108769212A (en) * | 2018-05-31 | 2018-11-06 | 康键信息技术（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111752960A (en) * | 2020-06-28 | 2020-10-09 | 北京百度网讯科技有限公司 | Data processing method and device |
| CN112114757A (en) * | 2020-09-28 | 2020-12-22 | 北京百度网讯科技有限公司 | Storage method and system in object storage system, computing device and medium |
Non-Patent Citations (3)
| Title |
|---|
| KUN QIAN; ZHONGZHI LUAN; ZUOWEI ZHANG; HAILONG YANG; YAQI YANG; DEPEI QIAN: "Data Analysis and Synchronization on Inter-Continent Data Placement Laboratory", IEEE * |
| 李明娟;邸海涛;张鹏;任晓瑞;: "云存储系统架构与关键技术研究", 信息通信, no. 03 * |
| 李锐;林艳萍;徐正全;冯蔚;: "空间数据存储对象的元数据可伸缩性管理", 计算机应用研究, no. 12 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115730016A (en) * | 2023-01-28 | 2023-03-03 | 北京国科天迅科技有限公司 | Data synchronization method, system, device, computer equipment and storage medium |
| CN115730016B (en) * | 2023-01-28 | 2023-09-19 | 北京国科天迅科技有限公司 | Data synchronization method, system, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860796B (en) | 2023-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113193947A (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| CN111338834A (en) | Data storage method and device | |
| CN113704058B (en) | Service model monitoring method and device and electronic equipment | |
| CN112948081B (en) | Method, device, equipment and storage medium for processing tasks in delayed mode | |
| CN112860796B (en) | Method, apparatus, device and storage medium for synchronizing data | |
| CN113067860A (en) | Method, apparatus, device, medium and product for synchronizing information | |
| CN115510036A (en) | Data migration method, device, equipment and storage medium | |
| CN113326038B (en) | Method, apparatus, device, storage medium and program product for providing service | |
| EP4092544A1 (en) | Method, apparatus and storage medium for deduplicating entity nodes in graph database | |
| CN113590447B (en) | Buried point processing method and device | |
| CN112887426B (en) | Information stream pushing method and device, electronic equipment and storage medium | |
| CN113641688B (en) | Node updating method, related device and computer program product | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN115639966A (en) | Data writing method and device, terminal equipment and storage medium | |
| CN115640280A (en) | Data migration method and device | |
| CN114722003A (en) | Centralized data acquisition method, device, equipment and storage medium | |
| CN114091909A (en) | Collaborative development method, system, device and electronic equipment | |
| CN114564149A (en) | Data storage method, device, equipment and storage medium | |
| CN113553216A (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN112925623A (en) | Task processing method and device, electronic equipment and medium | |
| CN113282489A (en) | Interface testing method and device | |
| CN113569144B (en) | Method, device, equipment, storage medium and program product for searching promotion content | |
| CN113360689B (en) | Image retrieval system, method, related device and computer program product | |
| US20230132173A1 (en) | Data reading method, device and storage medium | |
| CN113312521B (en) | Content retrieval method, device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |