CN112749228B - Data synchronization method, device, storage server and storage medium - Google Patents

Abstract

The invention discloses a data synchronization method, device, storage server and storage medium. Wherein, the method is applied to a first storage server, and the first storage server is used to store data blocks of the main block file. The method includes: sending a query request to the second storage server to query the first identification value in the second storage server. , the first identification value is the maximum identification value of the data block in the slave block file in the second storage server that is the same as the identification of the main block file; receive the first identification value sent by the second storage server; compare the first identification value with The second identification value is the maximum identification value of the data block in the main block file in the first storage server; when the first identification value is less than the second identification value, the second identification value of the second storage server is determined based on the first identification value. Missing data blocks from the chunk file; send the missing data blocks to the secondary storage server to synchronize the data blocks of the master chunk file and the data blocks of the slave chunk file.

Description

Data synchronization method, device, storage server and storage medium

Technical field

The present invention relates to the field of communication technology, and more specifically, to a data synchronization method, a data synchronization device, a storage server, and a computer-readable storage medium.

Background technique

In a distributed object storage system, multiple block files are stored on the storage server. Each block file has a unique block identification (Block id). And each block file corresponds to three copies (Replica), and the three copies include 1 master copy and 2 slave copies. Each copy is stored in a different storage server, and the corresponding block ID is the same.

When the storage server performs a write operation on one of the block files serving as the primary copy, it first writes the data blocks to be written into the block file of the primary copy. After the writing of the block file of the master copy is completed, the data block to be written is forwarded to the corresponding two storage servers where the block files of the slave copy are located. The data block to be written is written into the corresponding block file by the storage server where the block file of the slave copy is located. Based on this, the three copies have the same data, that is, the data of the three copies are synchronized.

However, since the storage server where the block file of the slave copy is located, when writing the data block to be written into the corresponding block file, the data to be written is first written into the memory and then run on the storage server. Under the influence of the operating system, the data blocks to be written in the memory are refreshed to the corresponding block files in the disk. Therefore, after the storage server where the block file of the slave copy is located is powered off, there is a possibility that the data in the memory of the storage server that has not been refreshed to the disk will be lost. Based on this, when the storage service is restarted, the data in the block file as the slave copy will not be the same as the data in the corresponding block file as the master copy, that is, the data in the three copies will be out of sync.

Contents of the invention

An object of the present invention is to provide a technical solution for a new data synchronization method.

According to a first aspect of the present invention, a data synchronization method is provided. The method is applied to a first storage server, and the first storage server is used to store data blocks of a main block file. The method includes:

Send a query request to the second storage server to query the first identification value in the second storage server. The first identification value is the slave block in the second storage server that has the same identification as the main block file. The maximum identification value of data blocks in the file;

Receive the first identification value sent by the second storage server;

Comparing the first identification value with the second identification value, the second identification value is the maximum identification value of the data block in the main block file in the first storage server;

When the first identification value is less than the second identification value, determine the data block lost from the block file of the second storage server according to the first identification value;

The missing data blocks are sent to the second storage server to synchronize the data blocks of the master chunk file and the data blocks of the slave chunk file.

Optionally, sending a query request to the second storage server to query the first identification value in the second storage server includes:

According to the first time interval, a query request is periodically sent to the second storage server to query the first identification value in the second storage server.

Optionally, periodically sending query requests to the second storage server according to the first time interval to query the first identification value in the second storage server, including:

Within the first preset time period, query requests are periodically sent to the second storage server according to the first time interval to query the first identification value in the second storage server.

Optionally, determining the data blocks lost from the block file of the second storage server according to the first identification value includes:

Determine the data blocks corresponding to all identification values in the main block file that are greater than the first identification value and less than and equal to the second identification value to be missing from the secondary block file of the second storage server. data block.

Optionally, sending the lost data blocks to the second storage server to synchronize the data blocks of the master block file and the data blocks of the slave block file includes:

Send data blocks corresponding to all identification values in the main block file that are greater than the first identification value and less than and equal to the second identification value to the second storage server to synchronize the data blocks of the main block file and the data blocks from the block file.

Optionally, before sending a query request to the second storage server to query the first identification value in the second storage server, the method further includes:

Determine whether a new data block is written in the main block file within the second preset time period;

If it does not exist, perform the step of sending a query request to the second storage server to query the first identification value in the second storage server.

Optionally, the method also includes:

In response to the triggering of the set event, the operation of determining whether a new data block is written in the main block file within a second preset time period is performed.

Optionally, the method also includes:

When a data block to be stored is received, the data block to be stored is saved into the memory;

According to the second time interval, the data blocks to be stored in the memory are regularly refreshed to the main block file, wherein in the main block file, each of the data blocks to be stored obtains a corresponding identification value;

as well as,

Forward the data block to be stored to the second storage server to save the data block to be stored in a slave block file of the second storage server, wherein in the slave block file In the file, each data block to be stored obtains a corresponding identification value.

According to a second aspect of the present invention, a data synchronization device is provided, including:

A query module, configured to send a query request to the second storage server to query the first identification value in the second storage server, where the first identification value is the identification of the main block file in the second storage server. The maximum identification value of the data block in the same slave block file;

A receiving module, configured to receive the first identification value sent by the second storage server;

A comparison module, configured to compare the first identification value with the second identification value, where the second identification value is the maximum identification value of the data block in the main block file in the first storage server;

Determining module, configured to determine the data blocks lost from the block file of the second storage server according to the first identification value when the first identification value is less than the second identification value;

A sending module, configured to send the lost data blocks to the second storage server to synchronize the data blocks of the master block file and the data blocks of the slave block file.

According to a third aspect of the present invention, a storage server is provided, including the data synchronization device as described in the second aspect; or, including:

A processor and a memory, the memory being used to store executable instructions, the instructions being used to control the processor to execute the method according to any one of the first aspects.

According to a fourth aspect of the present invention, a computer-readable storage medium is provided. The storage medium stores computer instructions. When the computer instructions in the storage medium are executed by a processor, any one of the first aspects is implemented. method described.

In this embodiment, the first storage server sends a query request to the second storage server to query the first identification value in the second storage server; after receiving the first identification value sent by the second storage server, it compares its own The second identification value and the first identification value. When the first identification value is less than the second identification value, it is determined that a data block is lost in the secondary block file in the second storage server. At this time, the data blocks lost from the block file sent by the second storage service are determined according to the first identification value. Finally, the determined data blocks are sent to the second storage server, and the second storage server writes the lost data blocks into the slave block file. In this way, even if the second storage server loses data blocks due to power failure and restart, the data in the slave block file in the second storage server can be synchronized in time with the data in the master block file in the first storage server.

Other features of the invention and its advantages will become apparent from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings.

Description of the drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.

Figure 1 is a block diagram of the hardware configuration of a storage server provided by an embodiment of the present invention;

Figure 2 is a flow chart of a data synchronization method provided by an embodiment of the present invention.

Figure 3 is a schematic structural diagram of a data synchronization device provided by an embodiment of the present invention;

Figure 4 is a schematic structural diagram of a storage server provided by an embodiment of the present invention.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these examples do not limit the scope of the invention unless otherwise specifically stated.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered a part of the specification.

In all examples shown and discussed herein, any specific values are to be construed as illustrative only and not as limiting. Accordingly, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters refer to similar items in the following figures, so that once an item is defined in one figure, it does not need further discussion in subsequent figures.

<hardware configuration>

Figure 1 is a block diagram of the hardware configuration of a storage server provided by an embodiment of the present invention.

The storage server 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), or the like. The memory 1200 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is capable of wired or wireless communication, for example. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, or the like. The input device 1600 may include, for example, a touch screen, a keyboard, or the like. The user can input/output voice information through the speaker 1700 and the microphone 1800.

Although multiple devices are shown for the storage server 1000 in Figure 1, the present invention may only involve some of the devices, such as the memory 1200 and the processor 1100 in the storage server 1000. The data synchronization provided by the present invention The method may, for example, be stored in the memory 1200 in the form of an executable instruction, and the processor 1100 calls the executable instruction to perform the data synchronization method of the present invention.

In the above description, skilled persons can design instructions according to the solutions disclosed in the present invention. How instructions control the processor to operate is well known in the art and will not be described in detail here.

<system>

The data synchronization method provided by the present invention is applied to distributed object storage systems. Among them, the distributed object storage system includes more than two storage servers. The disk of each storage server stores multiple fixed-length block files (usually with a capacity of GB level, and is a file in the physical sense). And each block file has a corresponding block identifier, and each block file corresponds to a role identifier. This role identifier is used to distinguish the block file from the main block file (that is, the block file that serves as the master copy) and slave block files (that is, block files that serve as slave copies). In addition, each block file is used to store objects uploaded by users. It can be understood that the object is a data block uploaded by the user. The data block can be a picture, a piece of text, etc.

For the main block file, the storage server where the main block file is located will write the received object to the specified main block file. After the writing is completed, the received object is synchronized to the storage server where the slave block file corresponding to the master block file is located. The storage server where the slave block file corresponding to the master block file is located writes the received object into the corresponding slave block file.

It should be noted that the block identifiers of the master block file and the slave block file are the same. The storage server where the master block file is located and the storage server where the slave block file corresponding to the master block file is located are independent of each other, and The objects in the master block file must be consistent with the objects in the corresponding slave block file.

In addition, distributed object storage systems usually include clients and control terminals. Among them, the client is used to receive data uploaded by users. In some implementations, the client can be used to split the data uploaded by the user into multiple data blocks (or objects) when the data is too large. The client can also be used for read operations, for example, sending a data read request to the storage server to read the corresponding data from the storage server to provide it to the user. The client can also be used for delete data operations.

The control terminal monitors the remaining storage capacity of the storage server's disk, the remaining storage capacity of the main block file in the storage server, and monitors whether the client uploads objects. At the same time, the control end will record the block identifiers of the master block file and slave block file stored in the disk of each storage server, as well as record the block identifiers of each master block file and the storage server where the corresponding slave block file is located. Server ID.

In addition, the process of data synchronization in a distributed object storage system is as follows:

Step 1: The user uploads the object through the client; the client receives the object uploaded by the user, and when the object is too large (for example, larger than the size of any block file), the object can be segmented. On the contrary, if it is smaller than the size of the block file, the object may not be split.

Step 2: When the control terminal detects that the client has an object uploaded by the user, it can select a storage server from all storage servers in the distributed object storage system. For example, the storage server can be selected based on the remaining storage capacity of the disk of the storage server. Server; after selecting the storage server, further select a main block file from the storage server as the main block file to store the object (or data block) uploaded by the user this time, example Optionally, the main block file that stores the object uploaded by the user this time can be selected based on the remaining storage capacity of each main block file in the memory (for example, select the main block file with the largest remaining storage capacity as the main block file that stores the user's current uploaded object). The main block file of the uploaded object); send the block ID of the selected main block file and the server ID of the storage server where the main block file is located to the client to instruct the client to upload the object uploaded by the user. Forwarded to the storage server corresponding to the server identifier (that is, the first storage server mentioned later in this embodiment), so that the storage server adds an object identifier to the object uploaded by the user, and adds the object identifier to the object Write to the main block file corresponding to the block ID. Among them, the object identifiers added by the storage server to the objects uploaded by the user in sequence increase continuously from small to large.

Step 3: When the storage server writes the object uploaded by the user into the corresponding main block file, the storage server forwards the written object and the block identifier of the main block file to the corresponding main block file. From the storage server where the block file is located (that is, the second storage server mentioned later in this embodiment). This causes the storage server where the slave block file is located to write the received data block into the slave block file with the same identifier as the received master block file. It should be noted that the storage server obtains in advance from the control terminal the server identifier of the storage server where the slave block file corresponding to each master block file is located.

In addition, in the process of writing objects to the corresponding block files in the storage server, the object is first written into the memory, and then the data in the memory is refreshed under the action of the operating system running on the storage server. to a block file on disk.

<Method Example>

As shown in Figure 2, this embodiment provides a data synchronization method, which is applied to a first storage server, and the first storage server is used to store data blocks of the main block file.

It can be understood that although the first storage server is also used to store data blocks of the slave block file, in this embodiment of the present invention, the first storage server is used to store data blocks of the main block file. Therefore, the main area The slave block file associated with the block file cannot be stored in the first storage server. In this application, the description of the first storage server and the second storage server is used to distinguish the storage server where the master block file and the slave block file of any block file are located. In addition, the above data blocks are objects in the distributed storage system.

Figure 2 is a flow chart of a data synchronization method provided by an embodiment of the present invention.

As shown in Figure 2, the method includes the following S2100-S2500:

S2100. Send a query request to the second storage server to query the first identification value in the second storage server. The first identification value is the data block in the slave block file in the second storage server that has the same identification as the main block file. The maximum identification value.

In one embodiment, the specific implementation of the above S2100 may be: sending a query request to the second storage server in real time to query the first identification value in the second storage server.

In another embodiment, the specific implementation of the above S2100 may be the following S2110:

S2110. According to the first time interval, periodically send a query request to the second storage server to query the first identification value in the second storage server.

In this embodiment, the first time interval can be set according to actual requirements. In one example, it can be set according to the actual data upload and update situation. For example, the first time interval can be any time from 1s to 60s. For example, the first time interval can be 1s, 5s, 10s, or 15s. , 30s, 1min any one. In addition, by regularly sending query requests to the second storage server according to the first time interval, the waste of computing resources of the first storage server can be reduced. In addition, the first storage server can be configured according to different applications and actual data upload and update requirements. The time interval can not only meet actual needs, but also reduce the waste of storage server computing resources.

It should be noted that the identifiers of each main block file and the data blocks written in the slave block file are consecutive. The data block corresponding to the largest identifier value among the identifier values of the data blocks is the latest written, and the largest The data blocks corresponding to the identification value before the identification value have been successfully written to the disk. In addition, under the action of the operating system, the first storage server refreshes the data blocks in the memory to the block file in the disk in order of the identification values of the data blocks from small to large.

In addition, when the second storage server is restarted after a power failure, the first storage server may not be able to receive data blocks when forwarding them to it, resulting in lost data blocks, and the identification values corresponding to the lost data blocks are continuous. For example, if the data blocks that have been written in the second storage server are data block 1, data block 2, data block 3, data block 4, ..., data block 20. Among them, "1", "2", "3", "4", ... "20" refer to the identification value of the data block. When the second storage server is powered off and restarted, the data blocks lost by the second storage server are data block 20, data block 19, and data block 18. Therefore, the identification value of the data block in the first storage server may be inconsistent with the identification value of the data block in the second storage server.

In one embodiment, the specific implementation of the above S2110 can be achieved through the following S2111:

S2111. Within the first preset time period, regularly send query requests to the second storage server according to the first time interval to query the first identification value in the second storage server.

In this embodiment, the first preset time period defined in the above-mentioned S2111 may be a time period during which the second storage server restarts to stable operation after power failure. In some implementations, the first preset time period may be preset based on experience values of technicians. In other embodiments, the time required for a large number of storage servers to be powered off and restarted until stable operation can be calculated, data modeling is performed, and then the time period predicted by the mathematical model is used as the first preset time period.

For example, assuming that the first preset time period is 1 minute and the first time interval is 15 seconds, within the first preset time period, the first storage server can send four query requests to the second storage server. It should be noted that in this example, the first preset time period and the first time interval are exemplary, and the present invention is not limited thereto.

In this embodiment, within the first preset time period, query requests are periodically sent to the second storage server according to the first time interval to query the first identification value in the second storage server. During the time period when there is a possibility of data loss in the second memory, the step of sending the query request is performed again. There is no need to perform the step of sending a query request after the second storage server runs stably and no more data is lost in the memory. This can avoid a waste of computing resources of the first storage server and the second storage server.

S2200. Receive the first identification value sent by the second storage server.

In this step, the first storage server receives the first identification value returned by the second storage server in response to the query request. For example, if the maximum identification value of the data blocks currently stored by the second storage server is 20, 20 will be returned to the first storage server as the first identification value.

S2300. Compare the first identification value and the second identification value. The second identification value is the maximum identification value of the data block in the main block file in the first storage server.

In this embodiment of the present invention, when no data loss occurs on the second storage server, the first identification value and the second identification value should be the same. However, if data loss occurs in the second storage server, the first identification value and the second identification value will be different. For example, the second identification value is greater than the first identification value.

In this step, the first storage server receives the first identification value returned by the second storage server. The first storage server will compare the second identification value of the data block it stores with the first identification value.

S2400. When the first identification value is less than the second identification value, determine the data blocks lost from the block file of the second server based on the first identification value.

In this embodiment, if the first identification value is less than the second identification value, it can be determined that the data block in the secondary block file in the second storage server is lost.

In this embodiment, the determination of the data blocks lost from the block file of the second server based on the first identification value in the above S2400 can be specifically implemented through the following S2410:

S2410. Determine the data blocks corresponding to all identification values in the main block file that are greater than the first identification value and less than and equal to the second identification value as data blocks lost from the secondary block file of the second storage server.

For example, if the first identification value is 20 and the second identification value is 25, it can be determined that the data blocks with identification values 21-25 in the second storage server are lost.

In other embodiments of the present invention, if the first identification value is equal to the second identification value, it can be determined that the data blocks in the secondary block file in the second storage server are not lost.

S2500. Send the lost data blocks to the second storage server to synchronize the data blocks of the master block file and the data blocks of the slave block file.

In this embodiment, after the first storage server sends the lost data blocks to the second storage server, the second storage server will rewrite the lost data blocks into the slave block file, thereby synchronizing the master block file. and data from chunk files.

In some embodiments, this step can be implemented by sending data blocks corresponding to all identification values greater than the first identification value and less than and equal to the second identification value in the main block file to the second storage server, thereby synchronizing the main The block file and its corresponding data blocks from the block file.

For example, if the first identification value is 20 and the second identification value is 25, then the data blocks corresponding to all identification values greater than the first identification value and less than and equal to the second identification value may include identification values 21, 22, 23, The data blocks corresponding to 24 and 25. In this step, the first storage server sends the data blocks corresponding to the identification values 21, 22, 23, 24 and 25 to the second storage server, so that the second storage server can The blocks are stored in the corresponding slave block files.

In this embodiment, the first storage server sends a query request to the second storage server to query the first identification value in the second storage server; after receiving the first identification value sent by the second storage server, it compares its own The second identification value and the first identification value. When the first identification value is less than the second identification value, it is determined that a data block is lost in the secondary block file in the second storage server. At this time, the data blocks lost from the block file sent by the second storage service are determined according to the first identification value. Finally, the determined data blocks are sent to the second storage server, and the second storage server writes the lost data blocks into the slave block file. In this way, even if the second storage server loses data blocks due to power failure and restart, the data in the slave block file in the second storage server can be synchronized in time with the data in the master block file in the first storage server.

In addition, in the present invention, the first storage server regularly sends query requests to the second storage server. On the one hand, it can save the computing resources of the first storage server and the second storage server. On the other hand, it can also discover the second storage server in time. Whether data is lost in the system, data can be synchronized in a timely manner, providing security guarantee for data storage in distributed storage systems.

In one embodiment, the data synchronization method provided by this embodiment also includes the following S2120 and S2130 before the above S2100:

S2120. Determine whether a new data block is written in the main block file within the second preset time period.

S2130. If it does not exist, perform the step of sending a query request to the second storage server to query the first identification value in the second storage server.

In this embodiment, if a new data block is written in the main block file within the first time period, at this time, the first storage server will forward the new data block to the second storage server. If data blocks are lost after the second storage server is powered off and restarted during the first period of time, in this case, when executing the above S2100, the first primary storage server will repeatedly send the above new data blocks to the second storage server. Therefore, in order to avoid the first primary storage server from repeatedly sending the new data block to the second storage server, the above-mentioned S2120 may be executed.

In one embodiment, based on the previous embodiment, the above S2120 can also be implemented through the following S2121:

S2121. In response to the triggering of the set event, perform an operation of determining whether a new data block is written in the main block file within the second preset time period.

In this embodiment, setting an event may be an event that instructs the first storage server to execute the data synchronization method including the above S2120 and S2130 provided in this embodiment. Based on this, in one example, a selection box is set on the display interface of the first storage server. When the selection box is in the selected state, that is, when the set event is triggered, the first storage server can execute the steps provided in this embodiment, including: The above data synchronization method of S2120 and S2130. Among them, the user's mouse click operation or touch operation can be detected to determine whether the selection box is in a selected state.

In this embodiment, the intelligence of the data synchronization method provided in this embodiment can be further improved.

In one embodiment, based on any of the above embodiments, the data synchronization method provided by this embodiment also includes the following S2610-S2630:

S2610. When receiving the data block to be stored, save the data block to be stored in the memory.

In this embodiment, the above-mentioned data blocks to be stored are data blocks uploaded by the client.

S2620. According to the second time interval, regularly refresh the data blocks to be stored in the memory to the main block file, where in the main block file, each data block to be stored obtains a corresponding identification value.

In one embodiment, the second time interval can be set according to actual needs. For example, the second time interval may be 10s, 20s, etc. In another embodiment, the second time interval may be a default time interval, such as the default 10s.

In this embodiment, when the first storage server receives the data block to be stored, or when the first storage server saves the received data block to be stored into the memory, the first storage server sends the data block to be stored to the memory. Block adds an identifier that corresponds to an identifier value. In this way, each data block to be stored will obtain a corresponding identification value.

It should be noted that after the first storage server refreshes the data blocks to be stored to the main block file, the data blocks to be stored will not be lost.

S2630. Forward the data block to be stored to the second storage server to save the data block to be stored in the slave block file of the second storage server. In the slave block file, each block file to be stored is The data block obtains the corresponding identification value.

In this embodiment, after the first storage server saves the data blocks to be stored into the memory, or after the first storage server refreshes the data blocks to be stored to the main block file, the first storage server will The data blocks to be stored are forwarded to the second storage server, so that the second storage server saves the received data blocks to be stored into the slave block file.

When the second storage server receives the data block to be stored, or when the second storage server saves the received data block to be stored, the second storage server adds an identifier to the data block to be stored, and the identifier corresponds to a Identity value. At this time, each data block to be stored obtains the corresponding identification value.

In one embodiment, when the second storage server saves the received data block to be stored to the slave block file, the process may be: store the received data block to be stored in the memory; according to the third time interval Periodically refresh the data blocks to be stored in the memory to the slave block file. In this way, the data block to be stored will not be lost. For the description of the third time interval, please refer to the description of the second time interval, which will not be described again here.

In this embodiment, when the first storage server receives the data block to be stored, it saves the data block to be stored in the memory. After that, the data blocks to be stored in the memory will be periodically refreshed to the main block file according to the second time interval. This can reduce the possibility of data block loss due to power outage and restart of the first storage server.

<Device Example>

As shown in Figure 3, the embodiment of the present invention also provides a data synchronization device 30. The server includes: a query module 31, a receiving module 32, a comparison module 33, a determination module 34, and a sending module 35; wherein:

Query module 31, configured to send a query request to the second storage server to query the first identification value in the second storage server, where the first identification value is the link between the second storage server and the main block file. The maximum identification value of data blocks in the slave block file that identifies the same;

Receiving module 32, configured to receive the first identification value sent by the second storage server;

The comparison module 33 is used to compare the first identification value with the second identification value, where the second identification value is the maximum identification value of the data block in the main block file in the first storage server;

Determining module 34, configured to determine the data blocks lost from the block file of the second storage server according to the first identification value when the first identification value is less than the second identification value;

The sending module 35 is configured to send the lost data blocks to the second storage server to synchronize the data blocks of the master block file and the data blocks of the slave block file.

In one embodiment, the sending module 35 is specifically used for:

According to the first time interval, a query request is periodically sent to the second storage server to query the first identification value in the second storage server.

In one embodiment, the sending module 35 is also specifically configured to regularly send query requests to the second storage server according to the first time interval within the first preset time period to query the second storage server. the first identification value.

In one embodiment, the determination module 34 is specifically configured to: determine the data blocks corresponding to all identification values in the main block file that are greater than the first identification value and less than and equal to the second identification value as the The data blocks lost from the block file of the second storage server.

In one embodiment, the sending module 35 is specifically configured to send data blocks corresponding to all identification values in the main block file that are greater than the first identification value and less than and equal to the second identification value to the third identification value. and two storage servers to synchronize the data blocks of the master block file and the data blocks of the slave block file.

In one embodiment, the query module 31 includes a determination unit and a query unit. Wherein, the determining unit is used to determine whether a new data block is written in the main block file within the second preset time period;

A query unit, configured to perform the step of sending a query request to the second storage server to query the first identification value in the second storage server if it does not exist.

In one embodiment, the data synchronization device 30 provided in this embodiment further includes a response module, wherein the response module is configured to, in response to the triggering of a set event, execute the determination of the main area within the second preset time period. Whether to write new data blocks into the block file.

In one embodiment, the data synchronization device 30 provided in this embodiment also includes a refresh module, which includes a saving unit and a refreshing unit.

Wherein, the saving unit is used to save the data block to be stored into the memory when receiving the data block to be stored;

A refresh unit, configured to regularly refresh the data blocks to be stored in the memory to the main block file according to a second time interval, wherein in the main block file, each of the data blocks to be stored The data block obtains the corresponding identification value;

The sending module is also used to forward the data block to be stored to the second storage server, so as to save the data block to be stored in the secondary block file of the second storage server, Wherein, in the slave block file, each data block to be stored obtains a corresponding identification value.

It should be noted that the specific implementation of each module in the device embodiment of the present invention can be found in the relevant content of the method embodiment of the present invention, and will not be described again here.

<storage server>

As shown in Figure 4, an embodiment of the present invention also provides a storage server 40. The storage server includes a memory 41 and a processor 42. in:

The memory 41 is used to store executable instructions; the instructions are used to control the processor 42 to execute the method according to any one of the above method embodiments.

The storage server 40 may be any storage server in the distributed object storage system.

<Computer-readable storage medium>

In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When executed by a processor, the computer program implements the method according to any one of the above method embodiments.

The invention may be a system, method and/or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement various aspects of the invention.

Computer-readable storage media may be tangible devices that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or Flash memory), Static Random Access Memory (SRAM), Compact Disk Read Only Memory (CD-ROM), Digital Versatile Disk (DVD), Memory Stick, Floppy Disk, Mechanical Coding Device, such as a printer with instructions stored on it. Protruding structures in hole cards or grooves, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or through electrical wires. transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing/processing devices, or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium in the respective computing/processing device .

Computer program instructions for performing operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source code or object code written in any combination of object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server implement. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through the Internet). connect). In some embodiments, by utilizing state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can Computer readable program instructions are executed to implement various aspects of the invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine that, when executed by the processor of the computer or other programmable data processing apparatus, , resulting in an apparatus that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause the computer, programmable data processing device and/or other equipment to work in a specific manner. Therefore, the computer-readable medium storing the instructions includes An article of manufacture that includes instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other equipment, causing a series of operating steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executed on a computer, other programmable data processing apparatus, or other equipment to implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that embody one or more elements for implementing the specified logical function(s). Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two consecutive blocks may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts. , or can be implemented using a combination of specialized hardware and computer instructions. It is well known to those skilled in the art that implementation through hardware, implementation through software, and implementation through a combination of software and hardware are all equivalent.

The embodiments of the present invention have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (11)

1. A data synchronization method, wherein the method is applied to a first storage server, and the first storage server is used for storing data blocks of a master block file, and the method comprises:

sending a query request to a second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file in the second storage server;

receiving the first identification value sent by the second storage server;

Comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server;

when the first identification value is smaller than the second identification value, determining the lost data block of the second storage server from the block file according to the first identification value;

transmitting the lost data block to the second storage server to synchronize the data block of the master block file and the data block of the slave block file;

the identification of each main block file and the identification of the data blocks written in the slave block file are continuous, the data block corresponding to the maximum identification value in the identification values of the data blocks is written in the latest, and the data block corresponding to the identification value before the maximum identification value is written in the magnetic disk successfully.

2. The method of claim 1, wherein the sending a query request to a second storage server to query a first identification value in the second storage server comprises:

and periodically sending a query request to a second storage server according to the first time interval so as to query the first identification value in the second storage server.

3. The method of claim 2, wherein periodically sending a query request to a second storage server at a first time interval to query a first identification value in the second storage server comprises:

and within a first preset time period, periodically sending a query request to a second storage server according to a first time interval so as to query a first identification value in the second storage server.

4. The method of claim 1, wherein said determining the lost data block from the block file for the second storage server based on the first identification value comprises:

and determining the data blocks which are in the main block file, are larger than the first identification value and smaller than all the identification values which are equal to the second identification value and correspond to the identification values as the data blocks lost from the block file of the second storage server.

5. The method of claim 4, wherein said sending the lost data block to the second storage server to synchronize the data block of the master block file and the data block of the slave block file comprises:

and sending the data blocks which are in the master block file, are larger than the first identification value and smaller than and equal to all the identification values of the second identification value to a second storage server so as to synchronize the data blocks of the master block file and the data blocks of the slave block file.

6. The method of claim 1, wherein prior to sending a query request to a second storage server to query a first identification value in the second storage server, the method further comprises:

determining whether a new data block is written in the main block file in a second preset time period;

and if the first identification value does not exist, the step of sending a query request to the second storage server to query the first identification value in the second storage server is executed.

7. The method of claim 6, wherein the method further comprises:

and responding to the trigger of the setting event, and executing the operation of determining whether to write a new data block in the main block file in a second preset time period.

8. The method according to claim 1, wherein the method further comprises:

when a data block to be stored is received, the data block to be stored is stored into a memory;

periodically refreshing the data blocks to be stored in the memory to the main block file according to a second time interval, wherein each data block to be stored in the main block file obtains a corresponding identification value;

The method comprises the steps of,

and forwarding the data blocks to be stored to the second storage server to store the data blocks to be stored into a slave block file of the second storage server, wherein each data block to be stored in the slave block file obtains a corresponding identification value.

9. A data synchronization device, comprising:

the query module is used for sending a query request to the second storage server to query a first identification value in the second storage server, wherein the first identification value is the maximum identification value of a data block in a slave block file, which is the same as the identification of a master block file, in the second storage server;

the receiving module is used for receiving the first identification value sent by the second storage server;

the comparison module is used for comparing the first identification value with a second identification value, wherein the second identification value is the maximum identification value of the data block in the main block file in the first storage server;

the determining module is used for determining the lost data block of the second storage server from the block file according to the first identification value when the first identification value is smaller than the second identification value;

A transmitting module, configured to transmit the lost data block to the second storage server, so as to synchronize the data block of the master block file and the data block of the slave block file;

the identification of each main block file and the identification of the data blocks written in the slave block file are continuous, the data block corresponding to the maximum identification value in the identification values of the data blocks is written in the latest, and the data block corresponding to the identification value before the maximum identification value is written in the magnetic disk successfully.

10. A storage server comprising the data synchronization device of claim 9; alternatively, it includes:

a processor and a memory for storing executable instructions for controlling the processor to perform the method according to any one of claims 1-8.

11. A computer readable storage medium, characterized in that the storage medium stores computer instructions, which, when executed by a processor, implement the method of any of claims 1-8.