CN109561120B - Small file backup method, system and management server - Google Patents
Small file backup method, system and management server Download PDFInfo
- Publication number
- CN109561120B CN109561120B CN201710884822.4A CN201710884822A CN109561120B CN 109561120 B CN109561120 B CN 109561120B CN 201710884822 A CN201710884822 A CN 201710884822A CN 109561120 B CN109561120 B CN 109561120B
- Authority
- CN
- China
- Prior art keywords
- backed
- file
- server
- storage
- backup
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The application provides a small file backup method, a system and a management server. The small file backup method provided by the application is applied to a small file backup system, and comprises the following steps: the method comprises the steps that a management server receives a backup request sent by a client aiming at a file to be backed up, selects a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, informs the client of the selected first gateway server, and informs the first gateway server of the selected plurality of storage servers; and after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server, and each storage server saves the file to be backed up after receiving the file to be backed up. According to the small file backup method, the small file backup system and the management server, file backup is carried out through the gateway server, and backup efficiency of small files can be improved.
Description
Technical Field
The present application relates to cloud storage technologies, and in particular, to a method and a system for backing up a small file, and a management server.
Background
At present, in order to improve the security of data, data is usually stored in multiple copies, and each copy is stored on a different storage server. In this way, as long as the data stored on any one storage server is not lost, the data stored on the storage server can be used for data recovery.
The existing file backup method mainly utilizes a main storage server to complete file backup. Specifically, when a file is backed up, the client sends the file to be backed up and the backup policy information (carrying the identifier of the backup storage server) to the main storage server, and the main storage server receives and stores the file to be backed up, records the backup task according to the backup policy information, executes the backup task, and sends the file to be backed up to each backup storage server to complete the file backup.
When the existing method is used for backing up small files (files smaller than 2M), on one hand, the main storage server needs to store the received files to be backed up, and on the other hand, the main storage server needs to search backup tasks to be executed from a large number of backup tasks (a task list records a large number of backup tasks, and the whole searching process is slow), then execute the searched backup tasks, read the files to be backed up from a storage area, and send the read files to be backed up to the backup storage server. This results in a slower and less efficient overall backup process.
Disclosure of Invention
In view of this, the present application provides a method, a system and a management server for small file backup, so as to solve the problem that the existing file backup method is low in efficiency when small file backup is performed.
In a first aspect, the present application provides a small file backup method, where the method is applied to a small file backup system, where the system includes a management server, a gateway server, and multiple storage servers, and the method includes:
the management server receives a backup request sent by a client aiming at a file to be backed up, selects a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, informs the client of the selected first gateway server and informs the first gateway server of the selected plurality of storage servers;
after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server;
and after receiving the files to be backed up, each storage server stores the files to be backed up.
The second aspect of the present application still provides a small file backup method, where the method is applied to a management server, and the method includes:
receiving a backup request sent by a client for a file to be backed up; wherein the backup request carries the number of backup copies;
selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, wherein the total number of the storage servers is equal to the backup number;
notifying the client of the selected first gateway server so that the client sends the file to be backed up to the first gateway server;
and notifying the selected storage servers to the first gateway server, so that the first gateway server sends the file to be backed up to each storage server notified by the management server when receiving the file to be backed up sent by the client.
A third aspect of the present application provides a small file backup system comprising a management server, a gateway server, and a plurality of storage servers, wherein,
the management server is used for receiving a backup request sent by a client aiming at a file to be backed up, selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, notifying the client of the selected first gateway server, and notifying the first gateway server of the selected plurality of storage servers;
the first gateway server is used for sending the file to be backed up to each storage server notified by the management server after receiving the file to be backed up sent by the client;
and each storage server is used for storing the files to be backed up after receiving the files to be backed up.
A fourth aspect of the present application provides a management server, comprising: a receiving module, a processing module and a sending module, wherein,
the receiving module is used for receiving a backup request sent by a client aiming at a file to be backed up; wherein the backup request carries the number of backup copies;
the processing module is used for selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, wherein the total number of the plurality of storage servers is equal to the backup number;
the sending module is configured to notify the client of the selected first gateway server, so that the client sends the file to be backed up to the first gateway server;
the sending module is further configured to notify the selected storage servers to the first gateway server, so that when the first gateway server receives the file to be backed up sent by the client, the file to be backed up is sent to each storage server notified by the management server.
According to the small file backup method, the small file backup system and the management server, after the management server selects the gateway server and the storage servers for backing up files to be backed up according to the backup request sent by the client, the management server notifies the client of the selected first gateway server and notifies the first gateway server of the selected storage servers, and after receiving the files to be backed up sent by the client, the first gateway server sends the files to be backed up to each storage server notified by the management server. Therefore, the file backup is carried out through the gateway server, on one hand, the gateway server does not need to store the received file to be backed up, on the other hand, the process of searching the backup task and the process of reading the file to be backed up do not exist, the gateway server can directly send the received file to be backed up to each storage server notified by the management server, and the backup efficiency can be improved. In addition, the resource consumption of the storage server can be reduced, and the writing capacity of the storage server is improved.
Drawings
FIG. 1 is a schematic diagram of a file backup system provided herein;
FIG. 2 is a flowchart of a first embodiment of a file backup method according to the present application;
FIG. 3 is a flowchart of a second embodiment of a file backup method of the present application;
FIG. 4 is a flowchart of a third embodiment of a file backup method of the present application;
FIG. 5 is a flowchart of a fourth embodiment of a file backup method of the present application;
FIG. 6 is a flowchart of a fifth embodiment of a file backup method according to the present application;
FIG. 7 is a flowchart of a sixth embodiment of a file backup method of the present application;
FIG. 8 is a flowchart of a seventh embodiment of a file backup method according to the present application;
fig. 9 is a hardware configuration diagram of a management server according to an exemplary embodiment of the present application;
fig. 10 is a schematic structural diagram of a management server according to a first embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
The application provides a small file backup method, a small file backup system and a management server, and aims to solve the problem that the existing file backup method is low in efficiency when small files are backed up.
Before the file backup method provided by the present application is introduced, the file backup system provided by the present application is briefly introduced. Fig. 1 is a schematic diagram of a file backup system provided in the present application. Referring to fig. 1, the file backup system provided in the present application includes: a client 1, a management server 2, a gateway server 3 and a plurality of storage servers 4. It should be noted that, in the embodiment shown in fig. 1, the file backup system includes two gateway servers 31 and 32 and three storage servers 41, 42, and 43. Where 31 and 32 are the identities of the gateway servers and 41, 42 and 43 are the identities of the storage servers. It should be noted that the file backup system may also include only one gateway server.
The technical solution of the present application will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 2 is a flowchart of a first embodiment of a file backup method according to the present application. The main execution body of the embodiment is a small file backup system. The method provided by the embodiment may include:
s201, a management server receives a backup request sent by a client aiming at a file to be backed up; and selecting a gateway server and a plurality of storage servers for backing up the files to be backed up according to the backup request, notifying the client of the selected first gateway server, and notifying the first gateway server of the selected plurality of storage servers.
Specifically, when a user wants to perform file backup, a backup request is input to the client, and accordingly, when the client receives the backup request input by the user, the backup request is sent to the management server, so that the management server selects a gateway server and a plurality of storage servers for backing up files to be backed up according to the backup request.
It should be noted that, when the small file backup system only includes one gateway server, the management server selects the gateway server as the gateway server for backing up the file to be backed up; when the small file backup system includes a plurality of gateway servers and the management server records performance parameters of each gateway server and each storage server, for example, performance parameters such as CPU occupancy rate and memory occupancy rate of each gateway server and each storage server. The management server can select a relatively idle gateway server according to the locally recorded performance parameters of each gateway server, and select a plurality of storage servers for backing up files to be backed up according to the locally recorded performance parameters of each storage server. It should be noted that the total number of the selected storage servers for backing up the file to be backed up is equal to the number of backup copies carried in the backup request. For example, the file backup system includes 2 gateway servers and 9 storage servers ( gateway servers 31 and 32; storage servers 41, 42, 43, 44, 45, 46, 47, 48, and 49, respectively), the number of backup copies carried in the backup request is 5, in an embodiment, the selected first gateway server is gateway server 31, and the selected storage servers are storage servers 41 to 45.
Further, after the management server selects the gateway server and the plurality of storage servers, the management server notifies the client of the selected first gateway server, so that the client sends the file to be backed up to the first gateway server after receiving the notification message. In addition, the selected storage servers need to be notified to the first gateway server, so that after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server, so as to implement file backup.
In this step, the IP addresses and port numbers of the selected plurality of storage servers may be notified to the first gateway server. With reference to the above example, the port numbers and the IP addresses of the storage servers 41 to 45 are notified to the first gateway server 31, so that the first gateway server 31 can send the file to be backed up to the storage servers 41 to 45 after receiving the file to be backed up sent by the client, so as to implement file backup.
S202, after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server.
In this step, a specific manner in which the first gateway server sends the file to be backed up to each storage server notified by the management server is not limited. For example, the first gateway server may send the file to be backed up to each storage server through the foreground thread; the files to be backed up can also be sent to each storage server through the background thread; or the file to be backed up may be sent to each storage server by a foreground thread and a background thread in combination, for example, one of the files to be backed up is sent to any one of the storage servers by the foreground thread, and the remaining files to be backed up are sent to the remaining storage servers by the background thread.
It should be noted that the Common Language Runtime (CLR) can distinguish two different types of threads: foreground threads and background threads. The main difference between the two is that the application program can exit only after running all foreground threads; for the background thread, the application program can be directly quitted without considering whether the background thread is completely run or not, and all the background threads can be automatically ended when the application program is quitted. In addition, for more introduction about the foreground thread and the background thread, reference may be made to the description in the prior art, and details are not described here.
And S203, after receiving the file to be backed up, each storage server stores the file to be backed up.
As can be seen from the foregoing embodiments, in the file backup method provided in this embodiment, after the management server selects the gateway server and the plurality of storage servers for backing up the file to be backed up according to the backup request sent by the client, the management server notifies the client of the selected first gateway server and notifies the first gateway server of the selected plurality of storage servers, and after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server. Therefore, the file backup is carried out through the gateway server, on one hand, the gateway server does not need to store the received file to be backed up, on the other hand, the process of searching the backup task and the process of reading the file to be backed up do not exist, the gateway server can directly send the received file to be backed up to each storage server notified by the management server, and the backup efficiency can be improved. In addition, the resource consumption of the storage server can be reduced, and the writing capacity of the storage server is improved.
It should be noted that, in practical applications, when the client receives a backup request input by a user, the client may obtain, from the management server, an identifier of each gateway server and an identifier of each storage server in the file backup system, and present the identifiers to the user, the user may select the gateway server and the plurality of storage servers for backing up files to be backed up by himself, and when the user has selected the gateway servers, the client notifies the user of the selected first gateway server of the plurality of storage servers. In this way, after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the client, so as to implement file backup.
Fig. 3 is a flowchart of a second embodiment of the file backup method of the present application. On the basis of the foregoing embodiment, the method provided by this embodiment, step S202, may include:
s301, the first gateway server copies N copies of the file to be backed up, where N is the total number of the plurality of storage servers notified by the management server.
In this step, with reference to the previous example, 5 copies of the file to be backed up are made.
S302, the first gateway server sends one file to be backed up to a designated storage server through a foreground thread; the designated storage server is one of the storage servers notified by the management server.
Specifically, the designated storage server is any one of the plurality of storage servers notified by the management server. With reference to the example in S201, for example, the storage server is designated as the storage server 41, in this step, one of the files to be backed up is sent to the storage server 41 through the foreground thread.
And S303, the first gateway server sends the remaining N-1 files to be backed up to the remaining N-1 storage servers through the background front process.
With reference to the examples in S201 and S301, in this step, the remaining 4 files to be backed up are sent to the storage server 42, the storage server 43, the storage server 44, and the storage server 45 through the background thread.
The embodiment shows that when the file to be backed up is sent to each storage server, the backup efficiency can be further improved by combining the foreground thread and the background thread. In addition, when the files to be backed up are sent to each storage server, a foreground thread is adopted to send one of the files to be backed up to the designated storage server, and a background thread is adopted to send the rest of the files to be backed up to the rest of the storage servers. Therefore, when receiving the successful response message returned by the appointed storage server, the first gateway server can return the response message with successful backup to the client, and the first gateway server does not need to wait until all the storage servers return the successful response message, and the data processing capacity of the client can be improved.
Optionally, in a possible implementation manner of the present application, after step S302, the method further includes:
and when a successful response message returned by the appointed storage server is received, the first gateway server returns a response message of successful backup to the client.
It should be noted that, when the storage server successfully receives the file to be backed up sent by the first gateway server, a successful response message is returned to the first gateway server. And when receiving a successful response message returned by the designated storage server, the first gateway server returns a response message of successful backup to the client (at this time, the client can receive the next backup request input by the user). Therefore, the response message of successful backup is not required to be returned to the client when the response messages of successful backup returned by all the storage servers are received, and the service processing capacity of the client can be improved.
Fig. 4 is a flowchart of a third embodiment of a file backup method according to the present application. On the basis of the foregoing embodiment, after step S202, the method provided in this embodiment may further include:
s401, the first gateway server receives a successful response message returned by the storage server.
As can be seen from the foregoing description, when the storage server successfully receives the file to be backed up sent by the first gateway server, a successful response message is returned to the first gateway server. It should be noted that the successful response message carries the identity of the storage server.
S402, the first gateway server determines that the storage server which sends the successful response message successfully backs up the file to be backed up.
With reference to the foregoing example, after the first gateway server sends the file to be backed up to the storage server 41, the storage server 42, the storage server 43, the storage server 44, and the storage server 45, for example, in this embodiment, the first gateway server receives a successful response message returned by the storage server 41, the storage server 42, and the storage server 44, in this step, it is determined that the storage server 41, the storage server 42, and the storage server 44 successfully back up the file to be backed up.
When the method provided by the present application is used to perform backup of small files (files smaller than 2M), the data size of the small files is small, and therefore, the failure rate is low.
S403, the first gateway server notifies the storage server that successfully backs up the file to be backed up to the management server.
In connection with the previous example, storage server 41, storage server 42 and storage server 44 are passed to the management server.
And S404, the management server compares the total number A of the received storage servers which successfully back up the file to be backed up with the total number B of the selected plurality of storage servers.
With reference to the foregoing example, the storage servers received by the management server to successfully backup the file to be backed up are the storage server 41, the storage server 42 and the storage server 44, that is, the total number a of the storage servers to successfully backup the file to be backed up is 3, and the total number B of the plurality of storage servers selected by the management server is 5.
S405, when the A is smaller than the B, the management server sends an indication message to at least one storage server which successfully backs up the file to be backed up so as to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
Specifically, when a is smaller than B, it indicates that there is a storage server that has unsuccessfully backed up the file to be backed up, and at this time, the management server may send an indication message to at least one storage server that has successfully backed up the file to be backed up, so as to indicate the storage server that has received the indication message to back up the file to be backed up to the storage server that has unsuccessfully backed up the file to be backed up, which is indicated by the indication message. In a specific implementation, first, according to the received storage server that successfully backs up the file to be backed up and the selected plurality of storage servers, the storage server that unsuccessfully backs up the file to be backed up is determined, and then an indication message is sent to at least one storage server that successfully backs up the file to be backed up, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that unsuccessfully backs up the file to be backed up, which is indicated by the indication message.
For example, when there is a storage server that has not successfully backed up the file to be backed up, an indication message may be sent to any storage server that has successfully backed up the file to be backed up, so as to indicate the storage server to back up the file to be backed up to the storage server that has not successfully backed up the file to be backed up. When there are a plurality of storage servers which have not successfully backed up the file to be backed up, an indication message may be sent to any one of the storage servers which have successfully backed up the file to be backed up to indicate the storage server to back up the file to be backed up to the storage server which has not successfully backed up the file to be backed up, or an indication message may be sent to a plurality of storage servers which have successfully backed up the file to be backed up. In connection with the foregoing example, in this step, for example, an instruction message may be sent to the storage server 41 to instruct the storage server 41 to backup the files to be backed up in the storage server 43 and the storage server 45, respectively, or an instruction message may be sent to the storage server 41 and the storage server 42 to instruct the storage server 41 to backup the files to be backed up in the storage server 43 and instruct the storage server 42 to backup the files to be backed up in the storage server 45, respectively.
As seen from the above embodiments, after sending the file to be backed up to each storage server, the first gateway server receives a successful response message returned by the storage server, further determining that the storage server sending the successful response message successfully backs up the file to be backed up, and the storage server which is determined to successfully back up the file to be backed up is informed to the management server, and the management server compares the total number A of the received storage servers which successfully back up the file to be backed up with the total number B of the selected plurality of storage servers, further, when A is smaller than B, an indication message is sent to at least one storage server which successfully backs up the file to be backed up, and the storage server receiving the indication message is indicated to back up the file to be backed up to the storage server which is indicated by the indication message and does not successfully back up the file to be backed up. Therefore, the files to be backed up can be successfully backed up in each storage server, and the backup task is successfully completed.
Fig. 5 is a flowchart of a fourth embodiment of the file backup method of the present application. On the basis of the foregoing embodiment, after step S202, the method provided in this embodiment may include:
and S501, the first gateway server receives a success response message returned by the storage server.
And S502, the first gateway server determines that the storage server which sends the successful response message successfully backs up the file to be backed up.
Specifically, for the specific implementation process and implementation principle of steps S501 and S502, reference may be made to the description of steps S401 and S402, which is not described herein again.
S503, the first gateway server compares the determined total number M of storage servers that successfully backup the file to be backed up with the total number N of the plurality of storage servers notified by the management server.
With reference to the example in S402, the storage servers that successfully back up the files to be backed up are determined to be the storage server 41, the storage server 42, and the storage server 44, and the total number M of the storage servers that successfully back up the files to be backed up is determined to be 3. Further, in combination with the example in the first embodiment, the plurality of storage servers notified by the management server are the storage servers 41 to 45, that is, the total number N of the plurality of storage servers notified by the management server is 5. In this step, M and N are compared, e.g., in this example, M is determined to be less than N by the comparison.
And S504, when the M is smaller than the N, the first gateway server sends an indication message to the determined at least one storage server which successfully backs up the file to be backed up so as to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
When M is smaller than N, it indicates that there is a storage server that has unsuccessfully backed up the file to be backed up, and at this time, the first gateway server may send an indication message to at least one determined storage server that has successfully backed up the file to be backed up, so as to indicate the storage server that has received the indication message to back up the file to be backed up to the storage server that has unsuccessfully backed up the file to be backed up, which is indicated by the indication message. In a specific implementation, first, according to the determined storage servers that successfully back up the file to be backed up and the plurality of storage servers notified by the management server, the storage server that unsuccessfully backs up the file to be backed up is determined, and then an indication message is sent to at least one of the determined storage servers that successfully back up the file to be backed up, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that unsuccessfully backs up the file to be backed up, which is indicated by the indication message.
For example, when there is a storage server that has not successfully backed up the file to be backed up, an indication message may be sent to any storage server that has successfully backed up the file to be backed up, so as to indicate the storage server to back up the file to be backed up to the storage server that has not successfully backed up the file to be backed up. When there are a plurality of storage servers which have not successfully backed up the file to be backed up, an indication message may be sent to any one of the storage servers which have successfully backed up the file to be backed up to indicate the storage server to back up the file to be backed up to the storage server which has not successfully backed up the file to be backed up, or an indication message may be sent to a plurality of storage servers which have successfully backed up the file to be backed up. In connection with the example in S402, in this step, for example, the first gateway server may send an instruction message to the storage server 41 to instruct the storage server 41 to backup the file to be backed up into the storage server 43 and the storage server 45, respectively, or send an instruction message to the storage server 41 and the storage server 42 to instruct the storage server 41 to backup the file to be backed up into the storage server 43 and instruct the storage server 42 to backup the file to be backed up into the storage server 45, respectively.
As can be seen from the above embodiment, after determining the storage servers that successfully back up the file to be backed up, the first gateway server compares the total number M of the storage servers that successfully back up the file to be backed up with the total number N of the plurality of storage servers notified by the management server, and further sends an indication message to at least one storage server that successfully backs up the file to be backed up when M is smaller than N, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that did not successfully back up the file to be backed up, where the storage server is indicated by the indication message. Therefore, the files to be backed up can be successfully backed up in each storage server, and the backup task is successfully completed.
Fig. 6 is a flowchart of a fifth embodiment of the file backup method of the present application. This embodiment describes a process of file backup by interaction between devices in the file backup system as shown in fig. 1. The method provided by the embodiment may include:
s601, the client sends a backup request input by a user for a file to be backed up to a management server; wherein, the backup request carries the backup number of copies.
And S602, the management server selects a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, wherein the total number of the plurality of storage servers is equal to the backup number.
S603, the management server notifies the selected first gateway server to the client.
S604, the management server notifies the selected storage servers to the first gateway server.
For specific implementation processes and implementation principles of steps S601 and S604, reference may be made to the description of step S201 in the embodiment, and details are not described herein again.
And S605, the client sends the file to be backed up to the first gateway server.
Specifically, after receiving the first gateway server notified by the management server, the client sends the file to be backed up to the first gateway server, so as to complete file backup through the first gateway server.
And S606, after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server.
And S607, after each storage server receives the file to be backed up, storing the file to be backed up.
Specifically, the specific implementation process and implementation principle of steps S606 and S607 may refer to the description of step S202 and step S203 in the embodiment, and are not described herein again.
And S608, the first gateway server receives a success response message returned by the storage server.
And S609, the first gateway server determines that the storage server which sends the successful response message successfully backs up the file to be backed up.
S610, the first gateway server notifies the storage server that successfully backs up the file to be backed up to the management server.
S611, the management server compares the total number a of the received storage servers that successfully backup the file to be backed up with the total number B of the selected plurality of storage servers.
And S612, when the A is smaller than the B, the management server sends an indication message to at least one storage server which successfully backs up the file to be backed up so as to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
For the specific implementation process and implementation principle of steps S608 and S612, reference may be made to the description of the third embodiment, which is not described herein again.
As can be seen from the above embodiments, after the management server selects the gateway server and the plurality of storage servers, the management server notifies the gateway server of the plurality of storage servers, so that when receiving the file to be backed up sent by the client, the gateway server sends the file to be backed up to each storage server notified by the management server. Therefore, file backup can be carried out through the gateway server, when the file backup is carried out through the gateway server, on one hand, the gateway server does not need to store the received file to be backed up, on the other hand, the process of searching the backup task does not exist, the process of reading the file to be backed up does not exist, and the gateway server can directly send the received file to be backed up to the storage server, so that the backup efficiency can be improved. In addition, the resource consumption of the storage server can be reduced, and the writing capacity of the storage server is improved.
Fig. 7 is a flowchart of a sixth embodiment of a file backup method according to the present application. The execution subject of the present embodiment is a management server. The method provided by the embodiment may include:
s701, receiving a backup request sent by a client for a file to be backed up; wherein, the backup request carries the backup number of copies.
When a user wants to perform file backup, a backup request is input to the client, and correspondingly, when the client receives the backup request input by the user, the backup request is sent to the management server, so that the management server selects the gateway server and the plurality of storage servers for backing up files to be backed up according to the backup request.
And S702, selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, wherein the total number of the storage servers is equal to the backup number.
The management server records performance parameters of each gateway server and each storage server in the file backup system, for example, performance parameters such as CPU occupancy and memory occupancy of each gateway server and each storage server. The management server can select a relatively idle gateway server according to the locally recorded performance parameters of each gateway server, and select a plurality of storage servers for backing up files to be backed up according to the locally recorded performance parameters of each storage server. It should be noted that the total number of the plurality of storage servers is equal to the number of backup copies carried in the backup request. For example, the file backup system includes 2 gateway servers and 9 storage servers ( gateway servers 31 and 32; storage servers 41, 42, 43, 44, 45, 46, 47, 48, and 49, respectively), the number of backup copies is 5, and in this step, the selected gateway server is gateway server 31, and the selected storage servers are storage servers 41 to 45.
And S703, notifying the selected first gateway server to the client so that the client sends the file to be backed up to the first gateway server.
After the management server selects the first gateway server for backing up the file to be backed up, the first gateway server is informed to the client, so that the client knows that the gateway server for backing up the file to be backed up is the first gateway server after receiving the informing message, and then sends the file to be backed up to the first gateway server.
S704, notifying the selected storage servers to the first gateway server, so that the gateway server sends the file to be backed up to each storage server notified by the management server after receiving the file to be backed up sent by the client.
In this step, the plurality of storage servers are notified to the first gateway server, so that after receiving the file to be backed up sent by the client, the first gateway server sends the file to be backed up to each storage server notified by the management server, so as to implement file backup.
Fig. 8 is a flowchart of a seventh embodiment of a file backup method according to the present application. The execution subject of the present embodiment is a management server. On the basis of the foregoing embodiment, the method provided in this embodiment may further include:
and S801, receiving the storage server which is notified by the first gateway server and successfully backs up the file to be backed up.
S802, comparing the total number A of the received storage servers which successfully back up the file to be backed up with the total number B of the selected plurality of storage servers.
And S803, when the A is smaller than the B, the management server sends an indication message to at least one storage server which successfully backs up the file to be backed up so as to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
Specifically, the specific implementation process and the implementation principle of each step in this embodiment may refer to the description of the third embodiment, and are not described herein again.
As can be seen from the above embodiment, the management server compares the total number a of the storage servers that have successfully backed up the file to be backed up with the total number B of the selected plurality of storage servers, and further sends an indication message to at least one storage server that has successfully backed up the file to be backed up when a is smaller than B, so as to indicate the storage server that has received the indication message to back up the file to be backed up to the storage server that has not successfully backed up the file to be backed up, where the storage server is indicated by the indication message. Therefore, the files to be backed up can be successfully backed up in each storage server, and the backup task is successfully completed.
With continued reference to fig. 1, the present application further provides a small file backup system, which includes a management server, a gateway server, and a plurality of storage servers, wherein,
the management server is used for receiving a backup request sent by a client aiming at a file to be backed up, selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, notifying the client of the selected first gateway server, and notifying the first gateway server of the selected plurality of storage servers;
the first gateway server is used for sending the file to be backed up to each storage server notified by the management server after receiving the file to be backed up sent by the client;
and each storage server is used for storing the files to be backed up after receiving the files to be backed up.
The system of this embodiment may be used to implement the technical solution of the method embodiment shown in fig. 1, and the implementation principle and the technical effect are similar, which are not described herein again.
Further, the first gateway server is specifically configured to copy N copies of the file to be backed up, send one of the files to be backed up to a designated storage server through a foreground thread, and send the remaining N-1 files to be backed up to the remaining N-1 storage servers through a background thread; wherein, the N is the total number of the plurality of storage servers notified by the management server, and the designated storage server is one of the storage servers notified by the management server.
Further, the first gateway server is further configured to, when receiving a successful response message returned by the specified storage server, return a response message indicating that the backup is successful to the client.
Further, the first gateway server is further configured to receive a successful response message returned by the storage server, determine that the storage server that sent the successful response message successfully backs up the file to be backed up, and notify the management server of the determined storage server that successfully backs up the file to be backed up;
the management server is further configured to compare the total number a of the received storage servers that successfully back up the file to be backed up with the total number B of the selected plurality of storage servers, and send an indication message to at least one storage server that successfully backs up the file to be backed up when a is smaller than B, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that does not successfully back up the file to be backed up, where the storage server is indicated by the indication message.
Further, the first gateway server is further configured to receive a successful response message returned by a storage server, determine that the storage server that sent the successful response message successfully backs up the file to be backed up, compare the total number M of the storage servers that successfully back up the file to be backed up with the total number N of the plurality of storage servers notified by the management server, and send an indication message to at least one storage server that successfully backs up the file to be backed up when M is smaller than N, so as to indicate the storage server that received the indication message to back up the file to be backed up to the storage server that did not successfully back up the file to be backed up, where the storage server that received the indication message indicates the storage server that did not successfully back up the file to be backed up.
In addition, corresponding to the foregoing method embodiments, the present application also provides embodiments corresponding to the management server.
The management server of the present application may be implemented by software, or may be implemented by hardware, or a combination of hardware and software. The software implementation is taken as an example, and as a logical device, the device is formed by reading corresponding computer program instructions in the memory into the memory through the processor for operation. From a hardware level, fig. 9 is a hardware structure diagram of a management server according to an exemplary embodiment of the present application. Referring to fig. 9, in addition to the memory 50, the processor 60 and the network interface 70 shown in fig. 9, the management server may further include other hardware according to its actual functions, which is not described in detail.
Referring to fig. 10, the management server provided in this embodiment may include: a receiving module 500, a processing module 600, and a transmitting module 700, wherein,
the receiving module 500 is configured to receive a backup request sent by a client for a file to be backed up; wherein the backup request carries the number of backup copies;
the processing module 600 is configured to select a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, where a total number of the plurality of storage servers is equal to the number of backup copies;
the sending module 700 is further configured to notify the selected storage servers to the first gateway server, so that when the first gateway server receives the file to be backed up sent by the client, the file to be backed up is sent to each storage server notified by the management server.
The management server of this embodiment may be configured to execute the technical solution of the method embodiment shown in fig. 7, and the implementation principle and the technical effect are similar, which are not described herein again.
Further, the receiving module 500 is further configured to receive the storage server that is notified by the first gateway server to successfully backup the file to be backed up;
the processing module 600 is further configured to compare the total number a of the received storage servers that successfully backup the file to be backed up with the total number B of the selected multiple storages;
the sending module 700 is further configured to send an indication message to at least one storage server that successfully backs up the file to be backed up when a is smaller than B, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that unsuccessfully backs up the file to be backed up, where the storage server is indicated by the indication message.
The management server of this embodiment may be configured to execute the technical solution of the method embodiment shown in fig. 8, and the implementation principle and the technical effect are similar, which are not described herein again.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (14)
1. A small file backup method is applied to a small file backup system, the system comprises a management server, a gateway server and a plurality of storage servers, and the method comprises the following steps:
the management server receives a backup request sent by a client aiming at a file to be backed up, selects a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, informs the client of the selected first gateway server so that the client sends the file to be backed up to the first gateway server after receiving the notification, and informs the first gateway server of the selected plurality of storage servers;
the first gateway server sends the file to be backed up to each storage server notified by the management server after receiving the file to be backed up sent by the client based on the notification of the first gateway server received from the management server;
and after receiving the files to be backed up, each storage server stores the files to be backed up.
2. The method of claim 1, wherein the first gateway server sends the file to be backed up to each storage server notified by the management server, and wherein the sending comprises:
the first gateway server copies the file to be backed up by N times, wherein N is the total number of the plurality of storage servers notified by the management server;
the first gateway server sends one file to be backed up to a designated storage server through a foreground thread; the designated storage server is one of the storage servers notified by the management server;
and the first gateway server sends the remaining N-1 files to be backed up to the remaining N-1 storage servers through a background thread.
3. The method of claim 2, wherein after the first gateway server sends one of the files to be backed up to the designated storage server via the foreground thread, the method further comprises:
and when a successful response message returned by the specified storage server is received, the first gateway server returns a response message of successful backup to the client.
4. The method of claim 1, wherein after the first gateway server sends the file to be backed up to each storage server notified by the management server, the method further comprises:
the first gateway server receives a successful response message returned by the storage server, and determines that the storage server which sends the successful response message successfully backs up the file to be backed up;
the first gateway server informs the management server of the determined storage server which successfully backs up the file to be backed up;
the management server compares the total number A of the received storage servers which successfully back up the file to be backed up with the total number B of the selected plurality of storage servers;
and when the A is smaller than the B, the management server sends an indication message to at least one storage server which successfully backs up the file to be backed up so as to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
5. The method according to claim 1, wherein after the first gateway server sends the file to be backed up to each of the storage servers notified by the management server, the method further comprises:
the first gateway server receives a successful response message returned by the storage server, and determines that the storage server which sends the successful response message successfully backs up the file to be backed up;
the first gateway server compares the total number M of the storage servers which are determined to successfully back up the file to be backed up with the total number N of the plurality of storage servers which are notified by the management server;
and when the M is smaller than the N, the first gateway server sends an indication message to the determined at least one storage server which successfully backs up the file to be backed up so as to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
6. A file backup method is applied to a management server and comprises the following steps:
receiving a backup request sent by a client for a file to be backed up; wherein the backup request carries the number of backup copies;
selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, wherein the total number of the storage servers is equal to the backup number;
notifying the client of the selected first gateway server, so that the client sends the file to be backed up to the first gateway server after receiving the notification;
and notifying the selected storage servers to the first gateway server, so that the first gateway server sends the file to be backed up to each storage server notified by the management server when receiving the file to be backed up sent by the client based on the notification of the first gateway server received from the management server notification.
7. The method of claim 6, further comprising:
receiving the storage server which is notified by the first gateway server to successfully backup the file to be backed up;
comparing the total number A of the received storage servers which successfully back up the file to be backed up with the total number B of the selected plurality of storages;
and when the A is smaller than the B, sending an indication message to at least one storage server which successfully backs up the file to be backed up to indicate the storage server which receives the indication message to back up the file to be backed up to the storage server which does not successfully back up the file to be backed up and is indicated by the indication message.
8. A small file backup system, characterized in that the system comprises a management server, a gateway server and a plurality of storage servers, wherein,
the management server is used for receiving a backup request sent by a client aiming at a file to be backed up, selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, notifying the client of the selected first gateway server, so that the client sends the file to be backed up to the first gateway server after receiving the notification, and notifying the first gateway server of the selected plurality of storage servers;
the first gateway server is used for sending the file to be backed up to each storage server notified by the management server after receiving the file to be backed up sent by the client based on the notification of the first gateway server received from the management server;
and each storage server is used for storing the files to be backed up after receiving the files to be backed up.
9. The system according to claim 8, wherein the first gateway server is specifically configured to copy N copies of the file to be backed up, send one of the files to be backed up to a designated storage server through a foreground thread, and send the remaining N-1 files to be backed up to the remaining N-1 storage servers through a background thread; wherein, the N is the total number of the plurality of storage servers notified by the management server, and the designated storage server is one of the storage servers notified by the management server.
10. The system of claim 9, wherein the first gateway server is further configured to return a response message indicating that the backup is successful to the client when receiving a successful response message returned by the designated storage server.
11. The system according to claim 8, wherein the first gateway server is further configured to receive a successful response message returned by the storage server, determine that the storage server that sent the successful response message successfully backs up the file to be backed up, and notify the management server of the determined storage server that successfully backs up the file to be backed up;
the management server is further configured to compare the total number a of the received storage servers that successfully back up the file to be backed up with the total number B of the selected plurality of storage servers, and send an indication message to at least one storage server that successfully backs up the file to be backed up when a is smaller than B, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that does not successfully back up the file to be backed up, where the storage server is indicated by the indication message.
12. The system according to claim 8, wherein the first gateway server is further configured to receive a successful response message returned by a storage server, determine that the storage server that sent the successful response message successfully backs up the file to be backed up, compare the determined total number M of the storage servers that successfully backed up the file to be backed up with the total number N of the plurality of storage servers notified by the management server, and send an indication message to the determined at least one storage server that successfully backs up the file to be backed up to indicate, when M is smaller than N, the storage server that received the indication message backs up the file to be backed up to the storage server that did not successfully back up the file to be backed up, which is indicated by the indication message.
13. A management server, comprising: a receiving module, a processing module and a sending module, wherein,
the receiving module is used for receiving a backup request sent by a client aiming at a file to be backed up; wherein the backup request carries the number of backup copies;
the processing module is used for selecting a gateway server and a plurality of storage servers for backing up the file to be backed up according to the backup request, wherein the total number of the plurality of storage servers is equal to the backup number;
the sending module is configured to notify the client of the selected first gateway server, so that the client sends the file to be backed up to the first gateway server after receiving the notification;
the sending module is further configured to notify the selected storage servers to the first gateway server, so that when the first gateway server receives the file to be backed up, which is sent by the client based on the notification of the first gateway server received from the management server notification, the file to be backed up is sent to each storage server notified by the management server.
14. The management server according to claim 13,
the receiving module is further configured to receive the storage server that is notified by the first gateway server that the file to be backed up is successfully backed up;
the processing module is further configured to compare the total number a of the received storage servers that successfully backup the file to be backed up with the total number B of the selected plurality of storages;
the sending module is further configured to send an indication message to at least one storage server that successfully backs up the file to be backed up when a is smaller than B, so as to indicate the storage server that receives the indication message to back up the file to be backed up to the storage server that unsuccessfully backs up the file to be backed up, where the storage server is indicated by the indication message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710884822.4A CN109561120B (en) | 2017-09-26 | 2017-09-26 | Small file backup method, system and management server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710884822.4A CN109561120B (en) | 2017-09-26 | 2017-09-26 | Small file backup method, system and management server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109561120A CN109561120A (en) | 2019-04-02 |
| CN109561120B true CN109561120B (en) | 2022-03-01 |
Family
ID=65862845
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710884822.4A Active CN109561120B (en) | 2017-09-26 | 2017-09-26 | Small file backup method, system and management server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109561120B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352916B (en) * | 2020-02-24 | 2023-11-03 | 杭州数梦工场科技有限公司 | Data storage method, system and storage medium based on NAS storage system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789855A (en) * | 2009-12-30 | 2010-07-28 | 北京天融信科技有限公司 | Server and double-link backup method thereof |
| CN105404645A (en) * | 2015-10-27 | 2016-03-16 | 北京乐动卓越科技有限公司 | File management method in file server system and file server system |
| CN106250270A (en) * | 2016-07-28 | 2016-12-21 | 四川新环佳科技发展有限公司 | A kind of data back up method under cloud computing platform |
| CN107015884A (en) * | 2016-01-28 | 2017-08-04 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6658589B1 (en) * | 1999-12-20 | 2003-12-02 | Emc Corporation | System and method for backup a parallel server data storage system |
| WO2004090722A1 (en) * | 2003-04-11 | 2004-10-21 | Star Softcomm Pte Ltd | Data isolation system and method |
| US20160277261A9 (en) * | 2006-12-29 | 2016-09-22 | Prodea Systems, Inc. | Multi-services application gateway and system employing the same |
| CN103580906B (en) * | 2012-08-09 | 2018-02-27 | 腾讯科技（深圳）有限公司 | A kind of method of data backup, system and server |
| US9152466B2 (en) * | 2013-03-13 | 2015-10-06 | Barracuda Networks, Inc. | Organizing file events by their hierarchical paths for multi-threaded synch and parallel access system, apparatus, and method of operation |
| CN104978351A (en) * | 2014-04-09 | 2015-10-14 | 中国电信股份有限公司 | Backup method of mass small files and cloud store gateway |
| CN104899116B (en) * | 2015-06-11 | 2019-06-04 | 北京京东尚科信息技术有限公司 | Method, source server, destination server and the system of data backup |
-
2017
- 2017-09-26 CN CN201710884822.4A patent/CN109561120B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789855A (en) * | 2009-12-30 | 2010-07-28 | 北京天融信科技有限公司 | Server and double-link backup method thereof |
| CN105404645A (en) * | 2015-10-27 | 2016-03-16 | 北京乐动卓越科技有限公司 | File management method in file server system and file server system |
| CN107015884A (en) * | 2016-01-28 | 2017-08-04 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
| CN106250270A (en) * | 2016-07-28 | 2016-12-21 | 四川新环佳科技发展有限公司 | A kind of data back up method under cloud computing platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109561120A (en) | 2019-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2695083B1 (en) | Cluster unique identifier | |
| US8635493B2 (en) | High availability system allowing conditionally reserved computing resource use and reclamation upon a failover | |
| US20160275123A1 (en) | Pipeline execution of multiple map-reduce jobs | |
| US20190140902A1 (en) | Centralized configuration data in a distributed file system | |
| CN109582213B (en) | Data reconstruction method and device and data storage system | |
| CN107153643B (en) | Data table connection method and device | |
| US9984139B1 (en) | Publish session framework for datastore operation records | |
| US20180364948A1 (en) | Data Processing Method, Related Device, and Storage System | |
| CN105205143A (en) | File storage and processing method, device and system | |
| US9906479B1 (en) | Storage management for ephemeral messages | |
| CN113282564B (en) | Data storage method, system, node and storage medium | |
| CN111881476A (en) | Object storage control method and device, computer equipment and storage medium | |
| CN109561120B (en) | Small file backup method, system and management server | |
| CN109889561A (en) | A kind of data processing method and device | |
| CN113014608A (en) | Flow distribution control method and device, electronic equipment and storage medium | |
| CN108173892B (en) | Cloud mirror image operation method and device | |
| CN107294781B (en) | Method and system for cluster configuration node failover | |
| US20160085638A1 (en) | Computer system and method of identifying a failure | |
| US10949322B2 (en) | Collecting performance metrics of a device | |
| CN110798492B (en) | Data storage method and device and data processing system | |
| CN114296891A (en) | Task scheduling method, system, computing device, storage medium and program product | |
| CN109634721B (en) | Method and related device for starting communication between virtual machine and host | |
| CN111405313B (en) | Method and system for storing streaming media data | |
| KR101793963B1 (en) | Remote Memory Data Management Method and System for Data Processing Based on Mass Memory | |
| CN108769123B (en) | Data system and data processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |