CN112749228B - Data synchronization method, device, storage server and storage medium - Google Patents

Data synchronization method, device, storage server and storage medium Download PDF

Info

Publication number
CN112749228B
CN112749228B CN201911051155.7A CN201911051155A CN112749228B CN 112749228 B CN112749228 B CN 112749228B CN 201911051155 A CN201911051155 A CN 201911051155A CN 112749228 B CN112749228 B CN 112749228B
Authority
CN
China
Prior art keywords
storage server
identification value
block
data
block file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911051155.7A
Other languages
Chinese (zh)
Other versions
CN112749228A (en
Inventor
黎海兵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingsoft Cloud Network Technology Co Ltd
Beijing Kingsoft Cloud Technology Co Ltd
Original Assignee
Beijing Kingsoft Cloud Network Technology Co Ltd
Beijing Kingsoft Cloud Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingsoft Cloud Network Technology Co Ltd, Beijing Kingsoft Cloud Technology Co Ltd filed Critical Beijing Kingsoft Cloud Network Technology Co Ltd
Priority to CN201911051155.7A priority Critical patent/CN112749228B/en
Publication of CN112749228A publication Critical patent/CN112749228A/en
Application granted granted Critical
Publication of CN112749228B publication Critical patent/CN112749228B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data synchronization method, a data synchronization device, a storage server and a storage medium. The method is applied to a first storage server, and the first storage server is used for storing the data blocks of the main block file, and the method comprises the following steps: sending a query request to a second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file, which is the same as the identification of a master block file, in the second storage server; receiving a first identification value sent by a second storage server; comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server; when the first identification value is smaller than the second identification value, determining a data block of the second storage server lost from the block file according to the first identification value; the lost data blocks are sent to a second storage server to synchronize the data blocks of the master and slave block files.

Description

Data synchronization method, device, storage server and storage medium
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data synchronization method, a data synchronization device, a storage server, and a computer readable storage medium.
Background
In a distributed object storage system, a plurality of block files are stored on a storage server. Each Block file has a unique Block identification (Block id). And each block file corresponds to three copies (Replica), the 3 copies containing 1 master copy and 2 slave copies. Each copy is stored in a different storage server and the corresponding block identifications are the same.
When the storage server performs writing operation on one of the block files serving as the main copy, the data block to be written is written into the block file of the main copy. After the writing of the block file of the master copy is completed, the data block to be written is forwarded to the corresponding two storage servers where the block files serving as the slave copies are located. And writing the data block to be written into the corresponding block file by a storage server in which the block file of the slave copy is located. Based on this, the three copies are made to have the same data, i.e., the data of the three copies are synchronized.
However, as the storage server where the block file of the slave copy is located, when writing the data block to be written into the corresponding block file, firstly, writing the data to be written into the memory, and then refreshing the data block to be written into the memory into the corresponding block file in the disk under the action of the operating system operated by the storage server. Therefore, when the storage server where the block file of the copy is located is powered down, there is a possibility that data in the memory of the storage server that is not available to be refreshed to the disk is lost. Based on this, when the storage service is restarted, the data in the block file as the slave copy will not be identical to the data in the corresponding block file as the master copy, i.e., the data of the three copies are not synchronized.
Disclosure of Invention
An object of the present invention is to provide a new data synchronization method.
According to a first aspect of the present invention, there is provided a data synchronization method applied to a first storage server for storing data blocks of a master block file, the method comprising:
sending a query request to a second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file, which is the same as the identification of the master block file, in the second storage server;
receiving the first identification value sent by the second storage server;
comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server;
when the first identification value is smaller than the second identification value, determining the lost data block of the second storage server from the block file according to the first identification value;
and sending the lost data block to the second storage server to synchronize the data block of the master block file and the data block of the slave block file.
Optionally, the sending a query request to the second storage server to query the second storage server for the first identification value includes:
and periodically sending a query request to a second storage server according to the first time interval so as to query the first identification value in the second storage server.
Optionally, the sending, at a first time interval, a query request to a second storage server periodically to query a first identification value in the second storage server includes:
and within a first preset time period, periodically sending a query request to a second storage server according to a first time interval so as to query a first identification value in the second storage server.
Optionally, the determining, according to the first identification value, the data block lost from the block file of the second storage server includes:
and determining the data blocks which are in the main block file, are larger than the first identification value and smaller than all the identification values which are equal to the second identification value and correspond to the identification values as the data blocks lost from the block file of the second storage server.
Optionally, the sending the lost data block to the second storage server to synchronize the data block of the master block file and the data block of the slave block file includes:
And sending the data blocks which are in the master block file, are larger than the first identification value and smaller than and equal to all the identification values of the second identification value to a second storage server so as to synchronize the data blocks of the master block file and the data blocks of the slave block file.
Optionally, before the sending a query request to the second storage server to query the first identification value in the second storage server, the method further includes:
determining whether a new data block is written in the main block file in a second preset time period;
and if the first identification value does not exist, the step of sending a query request to the second storage server to query the first identification value in the second storage server is executed.
Optionally, the method further comprises:
and responding to the trigger of the setting event, and executing the operation of determining whether to write a new data block in the main block file in a second preset time period.
Optionally, the method further comprises:
when a data block to be stored is received, the data block to be stored is stored into a memory;
periodically refreshing the data blocks to be stored in the memory to the main block file according to a second time interval, wherein each data block to be stored in the main block file obtains a corresponding identification value;
The method comprises the steps of,
and forwarding the data blocks to be stored to the second storage server to store the data blocks to be stored into a slave block file of the second storage server, wherein each data block to be stored in the slave block file obtains a corresponding identification value.
According to a second aspect of the present invention, there is provided a data synchronizing device comprising:
the query module is used for sending a query request to a second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file, which is the same as the identification of the master block file, in the second storage server;
the receiving module is used for receiving the first identification value sent by the second storage server;
the comparison module is used for comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data blocks in the main block file in the first storage server;
the determining module is used for determining the lost data block of the second storage server from the block file according to the first identification value when the first identification value is smaller than the second identification value;
And the sending module is used for sending the lost data block to the second storage server so as to synchronize the data block of the master block file and the data block of the slave block file.
According to a third aspect of the present invention there is provided a storage server comprising a data synchronisation device as described in the second aspect; alternatively, it includes:
a processor and a memory for storing executable instructions for controlling the processor to perform the method according to any one of the first aspects.
According to a fourth aspect of the present invention there is provided a computer readable storage medium storing computer instructions which, when executed by a processor, implement a method as in any of the first aspects.
In this embodiment, the first storage server sends a query request to the second storage server to query the first identification value in the second storage server; after receiving the first identification value sent by the second storage server, comparing the second identification value of the second storage server with the first identification value. And when the first identification value is smaller than the second identification value, determining that the data block in the slave block file in the second storage server is lost. And determining the lost data block from the block file, which is served by the second storage, according to the first identification value. And finally, the determined data block is sent to a second storage server, and the second storage server writes the lost data block into the slave block file. Thus, even if the second storage server loses the data block due to power-down restarting, the data in the slave block file in the second storage server can be synchronized with the data in the master block file in the first storage server in time.
Other features of the present invention and its advantages will become apparent from the following detailed description of exemplary embodiments of the invention, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a block diagram of a hardware configuration of a storage server according to an embodiment of the present invention;
fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a data synchronization device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a storage server according to an embodiment of the present invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless it is specifically stated otherwise.
The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of exemplary embodiments may have different values.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
< hardware configuration >
Fig. 1 is a block diagram of a hardware configuration of a storage server provided by an embodiment of the present invention.
The storage server 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and so forth. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, or the like. The memory 1200 includes, for example, ROM (read only memory), RAM (random access memory), nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 can perform wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display, a touch display, or the like. The input device 1600 may include, for example, a touch screen, keyboard, etc. A user may input/output voice information through the speaker 1700 and microphone 1800.
Although a plurality of devices are shown for the storage server 1000 in fig. 1, the present invention may relate only to some of the devices, for example, the memory 1200 and the processor 1100 in the storage server 1000, the method of data synchronization provided by the present invention may be stored in the memory 1200, for example, in the form of executable instructions, and the processor 1100 invokes the executable instructions to perform the method of data synchronization of the present invention.
In the above description, the skilled person may design instructions according to the disclosed solution. How the instructions control the processor to operate is well known in the art and will not be described in detail here.
< System >
The data synchronization method provided by the invention is applied to a distributed object storage system. Wherein the distributed object storage system comprises more than two storage servers. A plurality of fixed-length block files (typically, capacity GB level, and physically one file) are stored in the disk of each storage server. And each block file has a corresponding block identifier, and each block file has a role identifier for dividing the block file into a master block file (i.e., the block file as the master copy) and a slave block file (i.e., the block file as the slave copy). In addition, each block file is used to store objects uploaded by the user. It will be appreciated that the object is a block of data uploaded by the user. The data block may be a picture, a text, etc.
For the master block file, the storage server where the master block file is located writes the received object into the designated master block file. After the writing is finished, the received object is synchronized to a storage server where the slave block file corresponding to the master block file is located. And writing the received object into the corresponding slave block file by the storage server of the slave block file corresponding to the master block file.
It should be noted that, the block identifiers of the master block file and the slave block file are the same, the storage servers where the master block file is located, the storage servers where the slave block files corresponding to the master block file are independent of each other, and the objects in the master block file are required to be consistent with the objects in the corresponding slave block file.
In addition, a client and a control end are generally included in the distributed object storage system. The client is used for receiving data uploaded by a user. In some embodiments, the client may be configured to perform a slicing process, such as slicing into multiple data blocks (or objects), when the data uploaded by the user is too large. The client may also be used for a read operation, e.g., sending a data read request to the storage server to read the corresponding data from the storage server for provision to the user. The client may also be used to delete data operations.
The control end monitors the residual storage capacity of a disk of the storage server, the residual storage capacity of a main block file in the storage server and whether the client uploads an object or not. Meanwhile, the control end records the block identifiers of the master block file and the slave block file stored in the disk of each storage server, and records the server identifiers of the storage servers where each master block file and the corresponding slave block file are located.
In addition, the process of realizing data synchronization of the distributed object storage system is as follows:
step one, uploading an object by a user through a client; the client receives the object uploaded by the user, and when the object is too large (for example, larger than the size of any block file), the object can be segmented. Otherwise, if the size of the object is smaller than the size of the block file, the object may not be subjected to segmentation processing.
When the control end detects that the client has an object uploaded by a user, selecting one storage server from all storage servers of the distributed object storage system, for example, selecting the storage server based on the residual storage capacity of a disk of the storage server; after the storage server is selected, a main block file is further selected from the storage server as a main block file for storing the object uploaded by the user, so as to store the object (or the data block) uploaded by the user, for example, the main block file for storing the object uploaded by the user can be selected based on the residual storage capacity of each main block file in the storage (for example, the main block file with the largest residual storage capacity is selected as the main block file for storing the object uploaded by the user); and sending the block identifier of the selected main block file and the server identifier of the storage server where the main block file is located to the client so as to instruct the client to forward the object uploaded by the user to the storage server corresponding to the server identifier (namely the first storage server mentioned later in the embodiment), thereby adding the object identifier to the object uploaded by the user by the storage server, and writing the object added with the object identifier into the main block file corresponding to the block identifier. The object identifiers added by the objects uploaded to the user by the storage server in sequence are continuously increased from small to large.
And step three, after the storage server writes the object uploaded by the user into the corresponding master block file, the storage server forwards the written object and the block identifier of the master block file to the storage server where the slave block file corresponding to the master block file is located (i.e. a second storage server mentioned later in this embodiment). So that the slave block file is located in the storage server, and the received data block is written into the slave block file with the same identification as the received master block file. It should be noted that, the storage server obtains the server identifier of the storage server where the slave block file corresponding to each master block file is located from the control end in advance.
In addition, in the process of writing the object into the corresponding block file in the storage server, the object is firstly written into the memory, and then the data in the memory is refreshed into the block file in the disk under the action of the operating system operated by the storage server.
< method example >
As shown in fig. 2, the data synchronization method provided in this embodiment is applied to a first storage server, where the first storage server is configured to store data blocks of a master block file.
It will be appreciated that, although the first storage server is further configured to store the data blocks of the slave block file, in the embodiment of the present invention, the first storage server is configured to store the data blocks of the master block file, so that the slave block file associated with the master block file cannot be stored in the first storage server. In the present application, descriptions of the first storage server and the second storage server are used to distinguish a master block file and a slave block file of any block file from the storage server where the slave block file is located. In addition, the data block is an object in the distributed storage system.
Fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present invention.
As shown in fig. 2, the method includes the following S2100-S2500:
and S2100, sending a query request to the second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in the slave block file, which is the same as the identification of the master block file, in the second storage server.
In one embodiment, the specific implementation of S2100 may be: and sending a query request to the second storage server in real time to query the first identification value in the second storage server.
In another embodiment, the specific implementation of S2100 may be S2110 as follows:
s2110, periodically sending a query request to a second storage server according to a first time interval to query a first identification value in the second storage server.
In this embodiment, the first time interval may be set according to actual requirements. In one example, the first time interval may be any one of 1s-60s, for example, the first time interval may be any one of 1s, 5s, 10s, 15s, 30s, and 1min, which may be set according to the uploading and updating conditions of the actual data. In addition, by periodically sending the query request to the second storage server according to the first time interval, the waste of the operation resources of the first storage server can be reduced, and in addition, the first time interval is set according to different applications and the uploading and updating requirements of actual data, so that the waste of the operation resources of the storage server can be reduced while the actual requirements are met.
It should be noted that, the identifiers of the data blocks written in each master block file and each slave block file are continuous, the data block corresponding to the maximum identifier value in the identifier values of the data blocks is written recently, and the data block corresponding to the identifier value before the maximum identifier value is written in the disk successfully. In addition, under the action of an operating system, the first storage server refreshes the data blocks in the memory into the block files in the disk according to the sequence from small to large of the identification values of the data blocks.
In addition, when the second storage server is restarted after power failure, the first storage server cannot receive the data blocks when forwarding the data blocks to the second storage server, so that the lost data blocks are caused, and the identification values corresponding to the lost data blocks are continuous. For example, if the data block already written in the second storage server is data block 1, data block 2, data block 3, data blocks 4, … …, data block 20. Where "1", "2", "3", "4", … … "20" refer to the identification value of a data block. When the second storage server is powered down, the second storage server is restarted, and the lost data blocks are the data block 20, the data block 19 and the data block 18. Thus, the identification value of the data block in the first storage server may not coincide with the identification value of the data block in the second storage server.
In one embodiment, the specific implementation of S2110 may be implemented as follows S2111:
s2111, periodically sending a query request to a second storage server according to a first time interval within a first preset time period so as to query a first identification value in the second storage server.
In this embodiment, the first preset period of time defined in S2111 may be a period of time from when the second storage server is powered down to when the operation is stable. In some embodiments, the first preset time period may be preset by an empirical value of a technician. In other embodiments, the time from the power-down restart to the stable operation of the mass storage server may be counted, data modeling may be performed, and then a time period predicted by a mathematical model may be taken as the first preset time period.
For example, assuming that the first preset time period is 1min and the first time interval is 15s, the first storage server may send 4 query requests to the second storage server during the first preset time period. It should be noted that, in this example, the first preset period of time and the first time interval are both exemplary, which is not limited by the present invention.
In this embodiment, in a first preset period of time, periodically sending a query request to the second storage server at a first time interval, so as to query the first identification value in the second storage server, and executing the step of sending the query request again in a period of time when there is a possibility of data loss in the second memory. And the step of sending the query request is not required to be executed when the data in the memory is lost after the second storage server runs stably. This can avoid waste of the operation resources of the first storage server and the second storage server.
S2200, receiving a first identification value sent by the second storage server.
In this step, the first storage server receives a first identification value returned by the second storage server in response to the query request. For example, if the maximum identification value of the current stored data block of the second storage server is 20, the 20 is returned to the first storage server as the first identification value.
S2300, comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server.
In the embodiment of the present invention, in the case that the second storage server does not have data loss, the first identification value and the second identification value should be the same. However, if the second storage server experiences a data loss, the first identification value and the second identification value may be different, and the second identification value is, illustratively, greater than the first identification value.
In this step, the first storage server receives the first identification value returned by the second storage server. The first storage server compares the second identification value of the data block it stores with the first identification value.
And S2400, when the first identification value is smaller than the second identification value, determining the lost data block of the second server from the block file according to the first identification value.
In this embodiment, if the first identification value is smaller than the second identification value, it may be determined that the data block in the slave block file in the second storage server is lost.
In this embodiment, the determining, in S2400, the data block of the second server that is lost from the block file according to the first identification value may be specifically implemented as follows S2410:
S2410, determining the data blocks in the main block file, which correspond to all the identification values greater than the first identification value and less than or equal to the second identification value, as the data blocks lost from the block file of the second storage server.
For example, a first identification value of 20 and a second identification value of 25, it may be determined that data blocks in the second storage server with identification values of 21-25 are lost.
In other embodiments of the present invention, it may be determined that the data block in the slave block file in the second storage server is not lost if the first identification value is equal to the second identification value.
S2500, the lost data block is sent to a second storage server to synchronize the data block of the master block file and the data block of the slave block file.
In this embodiment, after the first storage server sends the lost data block to the second storage server, the second storage server rewrites the lost data block into the slave block file, thereby synchronizing the data of the master block file and the slave block file.
In some embodiments, this step may be implemented by sending, to the second storage server, the data blocks in the master block file that correspond to all the identification values greater than the first identification value and less than or equal to the second identification value, so as to synchronize the data blocks of the master block file and the slave block files corresponding thereto.
For example, if the first identifier value is 20 and the second identifier is 25, then the data blocks corresponding to all the identifier values greater than the first identifier value and less than or equal to the second identifier value may include the data blocks corresponding to the identifier values 21, 22, 23, 24, and 25, and in this step, the first storage server sends the data blocks corresponding to the identifier values 21, 22, 23, 24, and 25 to the second storage server, so that the second storage server stores the data blocks in the corresponding slave block files.
In this embodiment, the first storage server sends a query request to the second storage server to query the first identification value in the second storage server; after receiving the first identification value sent by the second storage server, comparing the second identification value of the second storage server with the first identification value. And when the first identification value is smaller than the second identification value, determining that the data block in the slave block file in the second storage server is lost. And determining the lost data block from the block file, which is served by the second storage, according to the first identification value. And finally, the determined data block is sent to a second storage server, and the second storage server writes the lost data block into the slave block file. Thus, even if the second storage server loses the data block due to power-down restarting, the data in the slave block file in the second storage server can be synchronized with the data in the master block file in the first storage server in time.
In addition, in the invention, the first storage server sends the query request to the second storage server regularly, so that on one hand, the operation resources of the first storage server and the second storage server can be saved, and on the other hand, whether the data are lost in the second storage server can be timely found, and further, the data synchronization can be timely carried out, and the safety guarantee is provided for the data storage of the distributed storage system.
In one embodiment, the data synchronization method provided in this embodiment further includes the following S2120 and S2130 before S2100 described above:
s2120, determining whether a new data block is written in the main block file in a second preset time period.
S2130, in the absence, performing the step of sending a query request to the second storage server to query the first identification value in the second storage server.
In this embodiment, if a new data block is written in the main block file in the first period, the first storage server forwards the new data block to the second storage server. If the second storage server loses the data block after power-down and restarting in the first period, the first main storage server repeatedly sends the new data block to the second storage server when executing the above S2100. Therefore, in order to avoid the first primary storage server from repeatedly transmitting the new data block to the second storage server, S2120 described above may be performed.
In one embodiment, on the basis of the above embodiment, the above S2120 may be further implemented by the following S2121:
s2121, responding to the trigger of the setting event, and executing the operation of determining whether to write a new data block in the main block file in a second preset time period.
The setting event in the present embodiment may be an event indicating the first storage server to execute, and the data synchronization method including S2120 and S2130 described above is provided in the present embodiment. Based on this, in one example, a selection box is set on the display interface of the first storage server, and when the selection box is in a selected state, that is, the setting event is triggered, the first storage server may execute the data synchronization method including S2120 and S2130 provided in this embodiment. Wherein, whether the selection box is in the selected state can be determined by detecting a mouse click operation or a touch operation from the user.
In this embodiment, the intelligence of the data synchronization method provided in this embodiment may be further improved.
In one embodiment, based on any one of the above embodiments, the data synchronization method provided in this embodiment further includes the following steps S2610-S2630:
s2610, when a data block to be stored is received, the data block to be stored is stored into a memory.
In this embodiment, the data block to be stored is a data block uploaded by the client.
S2620, periodically refreshing the data blocks to be stored in the memory to a main block file according to a second time interval, wherein each data block to be stored in the main block file obtains a corresponding identification value.
In one embodiment, the second time interval may be set according to actual requirements. Illustratively, the second time interval may be 10s, 20s, etc. In another embodiment, the second time interval may be a default time interval, such as a default of 10s.
In this embodiment, when the first storage server receives a data block to be stored, or when the first storage server saves the received data block to be stored to the memory, the first storage server adds an identifier to the data block to be stored, where the identifier corresponds to an identifier value. In this way, each data block to be stored will obtain a corresponding identification value.
It should be noted that, after the first storage server refreshes the data block to be stored to the main block file, the data block to be stored will not be lost.
S2630, forwarding the data blocks to be stored to a second storage server to store the data blocks to be stored in a slave block file of the second storage server, wherein each data block to be stored in the slave block file obtains a corresponding identification value.
In this embodiment, after the first storage server saves the data block to be stored in the memory, or after the first storage server refreshes the data block to be stored to the master block file, the first storage server forwards the data block to be stored to the second storage server, so that the second storage server saves the received data block to be stored to the slave block file.
When the second storage server receives the data block to be stored, or when the second storage server saves the received data block to be stored, the second storage server adds an identifier to the data block to be stored, and the identifier corresponds to an identifier value. At this time, each data block to be stored obtains a corresponding identification value.
In one embodiment, when the second storage server saves the received data block to be stored to the slave block file, the process may be: storing the received data block to be stored in a memory; and regularly refreshing the data blocks to be stored in the memory to the slave block file according to a third time interval. In this way, the data block to be stored will not be lost. The description of the third time interval may refer to the description of the second time interval, which is not repeated herein.
In this embodiment, when the first storage server receives the data block to be stored, the data block to be stored is saved to the memory. And then periodically refreshing the data blocks to be stored in the memory into the main block file according to a second time interval. This reduces the likelihood of data block loss due to a power down restart of the first storage server.
< device example >
As shown in fig. 3, the embodiment of the present invention further provides a data synchronization device 30, where the server includes: the device comprises a query module 31, a receiving module 32, a comparison module 33, a determination module 34 and a sending module 35; wherein:
the query module 31 is configured to send a query request to a second storage server, so as to query a first identifier value in the second storage server, where the first identifier value is a maximum identifier value of a data block in a slave block file in the second storage server, where the identifier of the slave block file is the same as that of the master block file;
a receiving module 32, configured to receive the first identification value sent by the second storage server;
a comparing module 33, configured to compare the first identification value with a second identification value, where the second identification value is a maximum identification value of a data block in the main block file in the first storage server;
A determining module 34, configured to determine, according to the first identification value, the data block lost from the block file of the second storage server when the first identification value is smaller than the second identification value;
and a sending module 35, configured to send the lost data block to the second storage server, so as to synchronize the data block of the master block file and the data block of the slave block file.
In one embodiment, the sending module 35 is specifically configured to:
and periodically sending a query request to a second storage server according to the first time interval so as to query the first identification value in the second storage server.
In one embodiment, the sending module 35 is further specifically configured to: and within a first preset time period, periodically sending a query request to a second storage server according to a first time interval so as to query a first identification value in the second storage server.
In one embodiment, the determining module 34 is specifically configured to: and determining the data blocks which are in the main block file, are larger than the first identification value and smaller than all the identification values which are equal to the second identification value and correspond to the identification values as the data blocks lost from the block file of the second storage server.
In one embodiment, the sending module 35 is specifically configured to send, to the second storage server, data blocks in the master block file that are corresponding to all identification values that are greater than the first identification value and less than or equal to the second identification value, so as to synchronize the data blocks of the master block file with the data blocks of the slave block file.
In one embodiment, the query module 31 includes a determination unit and a query unit. The determining unit is used for determining whether a new data block is written in the main block file in a second preset time period;
and the query unit is used for executing the step of sending a query request to the second storage server to query the first identification value in the second storage server under the condition that the query request does not exist.
In an embodiment, the data synchronization device 30 provided in this embodiment further includes a response module, where the response module is configured to perform the operation of determining whether to write a new data block in the main block file in the second preset time period in response to a trigger of the setting event.
In one embodiment, the data synchronization device 30 provided in this embodiment further includes a refresh module, where the refresh module includes a save unit and a refresh unit.
The storage unit is used for storing the data blocks to be stored into the memory when the data blocks to be stored are received;
the refreshing unit is used for periodically refreshing the data blocks to be stored in the memory to the main block file according to a second time interval, wherein each data block to be stored in the main block file obtains a corresponding identification value;
the sending module is further configured to forward the data blocks to be stored to the second storage server, so as to save the data blocks to be stored to a slave block file of the second storage server, where each data block to be stored obtains a corresponding identification value in the slave block file.
It should be noted that, the specific implementation manner of each module in the embodiment of the apparatus of the present invention may refer to the relevant content of the embodiment of the method of the present invention, which is not described herein again.
< storage Server >
As shown in fig. 4, the embodiment of the present invention further provides a storage server 40. The storage server includes a memory 41 and a processor 42. Wherein:
the memory 41 is used for storing executable instructions; the instructions are for controlling the processor 42 to perform a method according to any of the method embodiments described above.
Wherein the storage server 40 may be any storage server in a distributed object storage system.
< computer-readable storage Medium >
In this embodiment, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method according to any of the above-described method embodiments.
The present invention may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.
The foregoing description of embodiments of the invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (11)

1. A data synchronization method, wherein the method is applied to a first storage server, and the first storage server is used for storing data blocks of a master block file, and the method comprises:
sending a query request to a second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file in the second storage server;
receiving the first identification value sent by the second storage server;
Comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server;
when the first identification value is smaller than the second identification value, determining the lost data block of the second storage server from the block file according to the first identification value;
transmitting the lost data block to the second storage server to synchronize the data block of the master block file and the data block of the slave block file;
the identification of each main block file and the identification of the data blocks written in the slave block file are continuous, the data block corresponding to the maximum identification value in the identification values of the data blocks is written in the latest, and the data block corresponding to the identification value before the maximum identification value is written in the magnetic disk successfully.
2. The method of claim 1, wherein the sending a query request to a second storage server to query a first identification value in the second storage server comprises:
and periodically sending a query request to a second storage server according to the first time interval so as to query the first identification value in the second storage server.
3. The method of claim 2, wherein periodically sending a query request to a second storage server at a first time interval to query a first identification value in the second storage server comprises:
and within a first preset time period, periodically sending a query request to a second storage server according to a first time interval so as to query a first identification value in the second storage server.
4. The method of claim 1, wherein said determining the lost data block from the block file for the second storage server based on the first identification value comprises:
and determining the data blocks which are in the main block file, are larger than the first identification value and smaller than all the identification values which are equal to the second identification value and correspond to the identification values as the data blocks lost from the block file of the second storage server.
5. The method of claim 4, wherein said sending the lost data block to the second storage server to synchronize the data block of the master block file and the data block of the slave block file comprises:
and sending the data blocks which are in the master block file, are larger than the first identification value and smaller than and equal to all the identification values of the second identification value to a second storage server so as to synchronize the data blocks of the master block file and the data blocks of the slave block file.
6. The method of claim 1, wherein prior to sending a query request to a second storage server to query a first identification value in the second storage server, the method further comprises:
determining whether a new data block is written in the main block file in a second preset time period;
and if the first identification value does not exist, the step of sending a query request to the second storage server to query the first identification value in the second storage server is executed.
7. The method of claim 6, wherein the method further comprises:
and responding to the trigger of the setting event, and executing the operation of determining whether to write a new data block in the main block file in a second preset time period.
8. The method according to claim 1, wherein the method further comprises:
when a data block to be stored is received, the data block to be stored is stored into a memory;
periodically refreshing the data blocks to be stored in the memory to the main block file according to a second time interval, wherein each data block to be stored in the main block file obtains a corresponding identification value;
The method comprises the steps of,
and forwarding the data blocks to be stored to the second storage server to store the data blocks to be stored into a slave block file of the second storage server, wherein each data block to be stored in the slave block file obtains a corresponding identification value.
9. A data synchronization device, comprising:
the query module is used for sending a query request to the second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file, which is the same as the identification of a master block file, in the second storage server;
the receiving module is used for receiving the first identification value sent by the second storage server;
the comparison module is used for comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server;
the determining module is used for determining the lost data block of the second storage server from the block file according to the first identification value when the first identification value is smaller than the second identification value;
A transmitting module, configured to transmit the lost data block to the second storage server, so as to synchronize the data block of the master block file and the data block of the slave block file;
the identification of each main block file and the identification of the data blocks written in the slave block file are continuous, the data block corresponding to the maximum identification value in the identification values of the data blocks is written in the latest, and the data block corresponding to the identification value before the maximum identification value is written in the magnetic disk successfully.
10. A storage server comprising the data synchronization device of claim 9; alternatively, it includes:
a processor and a memory for storing executable instructions for controlling the processor to perform the method according to any one of claims 1-8.
11. A computer readable storage medium, characterized in that the storage medium stores computer instructions, which, when executed by a processor, implement the method of any of claims 1-8.
CN201911051155.7A 2019-10-31 2019-10-31 Data synchronization method, device, storage server and storage medium Active CN112749228B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911051155.7A CN112749228B (en) 2019-10-31 2019-10-31 Data synchronization method, device, storage server and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911051155.7A CN112749228B (en) 2019-10-31 2019-10-31 Data synchronization method, device, storage server and storage medium

Publications (2)

Publication Number Publication Date
CN112749228A CN112749228A (en) 2021-05-04
CN112749228B true CN112749228B (en) 2024-01-26

Family

ID=75641413

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911051155.7A Active CN112749228B (en) 2019-10-31 2019-10-31 Data synchronization method, device, storage server and storage medium

Country Status (1)

Country Link
CN (1) CN112749228B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5594863A (en) * 1995-06-26 1997-01-14 Novell, Inc. Method and apparatus for network file recovery
WO2008085869A2 (en) * 2007-01-07 2008-07-17 Apple Inc. Synchronization methods and systems
CN102656565A (en) * 2009-10-26 2012-09-05 亚马逊技术股份有限公司 Failover and recovery for replicated data instances
CN102843264A (en) * 2012-09-21 2012-12-26 中国航空无线电电子研究所 Control method of double hosts in high-speed serial bus network
EP2924953A1 (en) * 2014-03-25 2015-09-30 Thorsten Sprenger Method and system for encrypted data synchronization for secure data management
CN108304503A (en) * 2018-01-18 2018-07-20 阿里巴巴集团控股有限公司 A kind of processing method of data, device and equipment
CN109558421A (en) * 2018-10-29 2019-04-02 中国建设银行股份有限公司 Data processing method, system, device and storage medium based on caching
CN110059135A (en) * 2019-04-12 2019-07-26 阿里巴巴集团控股有限公司 A kind of method of data synchronization and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10366024B2 (en) * 2016-05-09 2019-07-30 International Business Machines Corporation Synchronous input/output computer system including hardware invalidation of synchronous input/output context

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5594863A (en) * 1995-06-26 1997-01-14 Novell, Inc. Method and apparatus for network file recovery
WO2008085869A2 (en) * 2007-01-07 2008-07-17 Apple Inc. Synchronization methods and systems
CN102656565A (en) * 2009-10-26 2012-09-05 亚马逊技术股份有限公司 Failover and recovery for replicated data instances
CN102843264A (en) * 2012-09-21 2012-12-26 中国航空无线电电子研究所 Control method of double hosts in high-speed serial bus network
EP2924953A1 (en) * 2014-03-25 2015-09-30 Thorsten Sprenger Method and system for encrypted data synchronization for secure data management
CN108304503A (en) * 2018-01-18 2018-07-20 阿里巴巴集团控股有限公司 A kind of processing method of data, device and equipment
CN109558421A (en) * 2018-10-29 2019-04-02 中国建设银行股份有限公司 Data processing method, system, device and storage medium based on caching
CN110059135A (en) * 2019-04-12 2019-07-26 阿里巴巴集团控股有限公司 A kind of method of data synchronization and device

Also Published As

Publication number Publication date
CN112749228A (en) 2021-05-04

Similar Documents

Publication Publication Date Title
CN109815028B (en) System, method, apparatus and computer storage medium for data synchronization
JP6629697B2 (en) Method and apparatus for extracting a virtual machine image file
CN108156205B (en) Message pushing method, client, server and programmable device
US11190586B2 (en) Cloud storage and synchronization of messages
US9875161B2 (en) Data replication across servers
CN109144785B (en) Method and apparatus for backing up data
US11188560B2 (en) Synchronizing object in local object storage node
CN111488243B (en) Backup and recovery method and device for MongoDB database, electronic equipment and storage medium
CN103475939A (en) Method, device and server for processing play record
CN112346926B (en) Resource state monitoring method and device and electronic equipment
CN111694857A (en) Method and device for storing resource data, electronic equipment and computer readable medium
US20130262601A1 (en) Method and device for sending and receiving emails
US20240031120A1 (en) System and method for automatically synchronizing responses to conditions on devices
CN108234551B (en) Data processing method and device
CN105162682A (en) Method for realizing regular message pushing in off-line state
US10938633B2 (en) Facilitating resilient and fault tolerant asynchronous messaging
CN114816225A (en) Management method and device of storage cluster, electronic equipment and storage medium
CN113672748A (en) Multimedia information playing method and device
CN109213955B (en) Data processing method and related equipment
US20180198837A1 (en) Streaming api subscription without loss of events
CN114637599A (en) Cloud resource management method and device, electronic equipment and readable storage medium
CN112822227B (en) Data synchronization method, device, equipment and storage medium of distributed storage system
CN112749228B (en) Data synchronization method, device, storage server and storage medium
CN107346217B (en) Cloud hard disk management method, cloud computing management platform and virtualization platform
WO2024061199A1 (en) Cloud clipboard implementation method, device, and system and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant