WO2024002349A1

WO2024002349A1 - File management method, server, storage node, file storage system, and client

Info

Publication number: WO2024002349A1
Application number: PCT/CN2023/104790
Authority: WO
Inventors: 程超
Original assignee: 中兴通讯股份有限公司
Priority date: 2022-07-01
Filing date: 2023-06-30
Publication date: 2024-01-04
Also published as: CN117370275A

Abstract

Provided in the present disclosure is a file management method used for a management server. The method comprises: determining a hash ribbon, which comprises a plurality of hash values; determining a hash node correspondence according to the hash ribbon and the total capacity of each storage node in a file storage system, wherein the hash node correspondence comprises a corresponding hash value of each storage node in the hash ribbon, the number of hash values corresponding to each storage node is proportional to the total capacity of each storage node, each hash value in the hash ribbon is allocated to each storage node, and each storage node corresponds to one or more hash values; and sending the hash node correspondence to each storage node. Further provided in the present disclosure are a file management method used for a storage node and a client, and a management server, the storage node, a file storage system, and the client.

Description

File management method, server, storage node, file storage system, client

Cross-references to related applications

This application claims priority from Chinese patent application No. 202210767064.9 submitted on July 1, 2022. The content of this Chinese patent application is incorporated herein by reference in its entirety.

Technical field

The present disclosure relates to the field of distributed storage technology, and in particular to a file management method, a management server, a storage node, a file storage system, and a client.

Background technique

With the development of cloud computing, artificial intelligence, intelligent security, industrial Internet and other technologies, big data technology has been deeply integrated into agriculture, medical care, transportation, finance, education, energy and other industries. As a result, the storage of massive data also places higher requirements on the security, stability, and efficiency of data storage systems. In order to meet the needs of massive data storage, distributed file storage systems are increasingly widely used.

However, in a distributed file storage system, file management takes longer and the system efficiency is low.

Contents of the invention

In a first aspect, an embodiment of the present disclosure provides a file management method for a management server of a file storage system. The file storage system includes a management server and a plurality of storage nodes. The method includes: determining a file including a plurality of hashes. The hash band of the value; determine the hash node correspondence according to the hash band and the total capacity of each storage node in the file storage system; the hash node correspondence includes each storage node in the hash Corresponding hash values in the hash band, the number of hash values corresponding to each storage node is proportional to the total capacity of each storage node, and each hash value in the hash band is allocated to each storage node; and Will The hash node correspondence is sent to each storage node.

In a second aspect, embodiments of the present disclosure provide a file management method for a first storage node of a file storage system. The file storage system includes a management server and a plurality of storage nodes. The method includes: receiving a response to the first storage node. A creation request for a target file; determining the first target hash value corresponding to the first target file according to the information of the first target file, and determining the corresponding first target hash value according to the preset hash node correspondence relationship. The first target storage node; the hash node correspondence includes the hash value corresponding to each storage node in the hash band, the hash band includes multiple hash values, and the hash value corresponding to each storage node The number of values is proportional to the total capacity of each storage node, and each hash value in the hash band is assigned to each storage node; the first storage node itself is the first target storage node, Then the first storage node locally stores the first target file; the first storage node itself is different from the first target storage node, then the first storage node locally stores the first target file, and Create a link to the first target file in the first target storage node.

In a third aspect, embodiments of the present disclosure provide a file management method for a client. The method includes: determining a first target hash value corresponding to the first target file according to the information of the first target file. The preset hash node correspondence determines the first target storage node corresponding to the first target hash value; the first target storage node is a storage node of a file storage system, and the file storage system includes a management server and multiple storage nodes, the hash node correspondence includes the hash value corresponding to each storage node in the hash band, the hash band includes multiple hash values, and the number of hash values corresponding to each storage node Proportional to the total capacity of each storage node, each hash value in the hash band is assigned to each storage node; the final target storage node is determined according to the first target storage node; the first target storage node The proportion of used capacity in the storage node is negatively correlated with the probability that the first target storage node is determined to be the final target storage node; and sending a creation request for the first target file to the final target storage node.

In a fourth aspect, embodiments of the present disclosure provide a management server, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is The processor executes, causing the processor to execute any file management method in the embodiment of the present disclosure.

In a fifth aspect, embodiments of the present disclosure provide a storage node, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is The processor executes, causing the processor to execute any file management method in the embodiment of the present disclosure.

In a sixth aspect, an embodiment of the present disclosure provides a file storage system, which includes a management server and a plurality of storage nodes; the management server is the management server of the embodiment of the present disclosure; the storage node is the storage node of the embodiment of the present disclosure. .

In a seventh aspect, embodiments of the present disclosure provide a client, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor. Execution causes the processor to execute any file management method in the embodiments of the present disclosure.

Description of drawings

Figure 1 is a flow chart of a method for file management of a management server provided by an embodiment of the present disclosure;

Figure 2 is a flow chart of a method for file management of a storage node provided by an embodiment of the present disclosure;

Figure 3 is a flow chart of a method for client file management provided by an embodiment of the present disclosure;

Figure 4 is a block diagram of a management server provided by an embodiment of the present disclosure;

Figure 5 is a block diagram of a storage node provided by an embodiment of the present disclosure;

Figure 6 is a block diagram of a file storage system provided by an embodiment of the present disclosure;

Figure 7 is a block diagram of a client provided by an embodiment of the present disclosure;

Figure 8 is a schematic diagram of hash band allocation in a file management method provided by an embodiment of the present disclosure;

Figure 9 is a schematic diagram of the logical process of a storage node accessing a file storage system in a file management method provided by an embodiment of the present disclosure;

Figure 10 is a schematic structural diagram of a file link in a file management method provided by an embodiment of the present disclosure;

Figure 11 shows the creation of files in a file management method provided by an embodiment of the present disclosure. Schematic diagram of the client's logical process;

Figure 12 is a schematic diagram of the logical process of a storage node when creating a file in a file management method provided by an embodiment of the present disclosure; and

FIG. 13 is a schematic diagram of the logical process of reading and writing files in a file management method provided by an embodiment of the present disclosure.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the file management method, management server, storage node, file storage system, and client provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, but the embodiments shown may be embodied in different forms and the disclosure should not be construed as limited to the embodiments set forth below. These embodiments are provided so that this disclosure will be thorough and complete, and will fully understand the scope of the disclosure to those skilled in the art.

The drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure and form a part of the specification. They are used to explain the present disclosure together with the detailed embodiments and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed embodiments with reference to the accompanying drawings.

The present disclosure may be described with reference to plan and/or cross-sectional illustrations, which are schematic illustrations of the disclosure. Accordingly, example illustrations may be modified based on manufacturing techniques and/or tolerances.

The embodiments of the present disclosure and the features in the embodiments may be combined with each other without conflict.

The terminology used in this disclosure is for describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure, the term "and/or" includes any and all combinations of one or more of the associated listed items. As used in this disclosure, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used in this disclosure, the terms “comprising” and “made of” specify the presence of particular features, integers, steps, operations, elements and/or components but do not exclude the presence or the possibility that one or more other features, Integers, steps, operations, elements, components and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientific terms) have the same meanings as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be construed to have meanings consistent with their meanings in the context of the relevant art and the present disclosure, and will not be construed as having idealized or excessively formal meanings, except as provided herein. Disclosure is expressly so qualified.

The present disclosure is not limited to the embodiments shown in the drawings but includes modifications of configurations formed based on manufacturing processes. Accordingly, the regions illustrated in the figures are of a schematic nature and the shapes of the regions shown in the figures are illustrative of the specific shapes of regions of the element and are not intended to be limiting.

Distributed file storage systems of some related technologies include a file distribution server and multiple storage nodes, and file management is performed by the file distribution server. That is, the file distribution server randomly distributes the files that need to be created (written) to each storage node. And the randomness is used to ensure the balance of the capacity of each storage node.

However, all file management in the above method is undertaken by the file distribution server, which results in heavy network load and long file distribution delay, which affects the efficiency of the entire system.

In a first aspect, embodiments of the present disclosure provide a file management method for use in a management server of a file storage system. The file storage system includes a management server and multiple storage nodes.

The method of the embodiment of the present disclosure is executed by the management server in the file storage system.

The file storage system in the embodiment of the present disclosure includes multiple storage nodes for actually storing files, so it belongs to a "distributed file storage system"; the file storage system also includes a management server with a management function. However, the file storage system Management of files is not entirely performed by the management server.

The client (such as a user terminal, or an interface device of the file storage system) can interact with the file storage system, thereby performing file management through the file storage system.

FIG. 1 is a flow chart of a method for file management of a management server provided by an embodiment of the disclosure. Referring to FIG. 1 , the method of the embodiment of the disclosure includes the following steps S101 to S303.

In step S101, a hash strip including a plurality of hash values is determined.

The management server calculates a hash band to be used, and the hash band includes a plurality of hash values.

In step S102, according to the hash band and the data of each storage node in the file storage system, The total capacity determines the hash node correspondence.

The hash node correspondence includes the hash value corresponding to each storage node in the hash band. The number of hash values corresponding to each storage node is proportional to the total capacity of each storage node. Each hash value in the hash band A hash value is assigned to each storage node.

All hash values of the above hash bands are assigned to each storage node currently in the file storage system. Each hash value is assigned to each storage node, and each storage node corresponds to one or more hashes. value, and the number of hash values corresponding to each storage node is proportional to its total capacity (including used capacity and unused capacity).

That is to say, the hash value is allocated according to the proportion of the total capacity of each storage node to the total capacity of all storage nodes: the proportion (ratio) of the total capacity of each storage node to the total capacity of all storage nodes is equal to each The ratio of the number of hash values corresponding to each storage node to the number of all hash values in the hash band.

In step S103, the hash node correspondence is sent to each storage node.

The correspondence between storage nodes and hash values (correspondence between hash nodes) obtained after the above allocation is allocated to all storage nodes, so that each storage node knows the correspondence between all storage nodes and hash values.

In some embodiments, the total number of hash values in the hash strip is determined based on the maximum number of storage nodes that the file storage system can support.

As an embodiment of the present disclosure, the total number of hash values in the hash strip may be determined based on the maximum number of storage nodes N that the file storage system can support, for example, the length of the hash strip (the number of hash values ) can be 2 ^N , that is, the hash value is in [0,2 ^N -1].

Therefore, when a new storage node is added to the file storage system or a storage node is deleted, the total hash band remains unchanged, but the storage node corresponding to the specific hash value may change.

In some embodiments, the method of the embodiment of the present disclosure also includes: when the storage node changes in the file storage system, determine the updated hash node based on the hash band and the total capacity of each storage node in the file storage system after the change. Correspondence.

In some embodiments, storage node changes include accessing new storage nodes and/or deleting existing storage nodes.

Send the updated hash node correspondence to each storage node.

As a way of the embodiment of the present disclosure, when the file storage system has a storage node When changes occur (new storage nodes are connected or storage nodes are deleted), the proportion of the total capacity of the storage nodes will obviously change. Therefore, the management server needs to redistribute hash values at this time (but does not need to recalculate the hash band). And send the obtained "new" hash node correspondence to the storage node.

It should be understood that since each updated hash node correspondence will be sent to the storage node, so for the storage node, the "old" hash node correspondence before a certain update is actually the "new" hash of the last update. Node correspondence, so the correspondence between the storage node and the hash node before and after the update is known.

In a second aspect, embodiments of the present disclosure provide a file management method for the first storage node of a file storage system. The file storage system includes a management server and multiple storage nodes.

The method of the embodiment of the present disclosure is executed by a storage node in the file storage system.

It should be understood that the first storage node is only an example of a storage node that performs the file management method, and does not refer to any sequence. Any storage node in the file storage system can perform the method of the disclosed embodiment, that is, this The methods of the disclosed embodiments are general methods executable by all storage nodes.

FIG. 2 is a flow chart of a method for file management of a storage node provided by an embodiment of the present disclosure. Referring to FIG. 2 , the method of the embodiment of the present disclosure includes the following steps S201 to S204.

In step S201, a creation request for a first target file is received.

A certain storage node is selected as the final target storage node (for example, the storage node that actually stores the content of the first target file), and thus a request is received to create (for example, write) a file (for example, the first target file) in the storage node. a target file).

It should be understood that the first target file, the second target file, and the third target file in the embodiments of the present disclosure do not refer to specific files, but are determined according to the application scenario, that is, the same file may be of different types in different application scenarios. target file.

For example, when a file is written to the file storage system, the file is the first target file; and after writing, when the file is opened, deleted, etc., the file is the second target file.

In step S202, the corresponding first target is determined based on the information of the first target file. The hash value determines the first target storage node corresponding to the first target hash value according to the preset hash node correspondence relationship.

In some embodiments, the hash node correspondence includes the hash value corresponding to each storage node in the hash band, the hash band includes multiple hash values, and the number of hash values corresponding to each storage node is proportional to The total capacity of each storage node, each hash value in the hash band is assigned to each storage node, and each storage node corresponds to one or more hash values.

The storage node can calculate the corresponding first hash value based on part of the information (such as the file name) of the first target file. The first hash value must be located in the above hash band, so the storage node can pre-store the file based on the first hash value in the storage node. The hash node correspondence relationship (for example, sent by the management server) determines the first target storage node corresponding to the first hash value, that is, the storage node where the first target file should theoretically be located.

In some embodiments, the first target storage node may be the first storage node, that is, the storage node itself that executes the method, or it may be another storage node. Different subsequent operations may be performed depending on the situation.

In step S203, if the first storage node, that is, the storage node itself that executes the method, is the first target storage node, the first storage node locally stores the first target file.

If the storage node that executes this method is itself the first target storage node, it means that the first target storage node is also the final target storage node (the storage node that actually stores the content of the first target file), so the first target file can be directly stored locally.

In step S204, if the first storage node, that is, the storage node itself that executes the method, is different from the first target storage node, the first storage node locally stores the first target file and creates the first target file in the first target storage node. A link to an object file.

If the storage node that executes this method is not the first target storage node, it means that the first target storage node is different from the final target storage node; therefore, the storage node can store the content of the first target file locally, but it must also store the content of the first target file locally. A link to the first target file is created in the first target storage node, so that the client can find the link to the first target file in the first target storage node and operate the first target file through the first target storage node.

In some embodiments, the method of the embodiment of the present disclosure further includes: receiving an operation request for the second target file.

In some embodiments, the operation includes opening the second target file, and/or deleting the second target file.

When the second target file is stored locally, the second target file is operated according to the operation request.

When the second target file is not stored locally, the operation request is responded to based on the link of the second target file.

After a file is written to a storage node (eg, the first target storage node), it may also receive a request to delete, open (read), and other operations on the file (eg, the second target file) stored therein.

At this time, if the second target file is stored locally on the storage node (that is, the storage node is also the final target storage node), the file can be directly operated; and if there is no second target file locally on the storage node (that is, the storage node node is not the final target storage node), obtain the link to the second target file, and respond to the above operations based on the link (for example, open the file corresponding to the link, or delete the file corresponding to the link and the link itself).

In some embodiments, the method of the embodiment of the present disclosure further includes: receiving updated hash node correspondence.

The corresponding third target hash value is determined according to the locally stored third target file information, and the third target storage node corresponding to the third target hash value is determined according to the updated hash node correspondence.

If the storage node itself that executes this method is different from the third target storage node, then the third target file is migrated to the third target storage node.

When a storage node changes in the file storage system (for example, a new storage node is connected or a storage node is deleted), each storage node will receive the "new" hash node correspondence assigned by the management server, and the "new" hash node In the node correspondence relationship, the storage nodes corresponding to some hash values may be different from before, so the storage nodes where the files with corresponding hash values should be located may also be different from before.

To this end, each storage node needs to check the file it stores (for example, the third target file), and according to the storage node (for example, the third target storage node) where the "new" hash node correspondence should be: if the third target storage node If the third target storage node is the storage node itself, no operation is required; if the third target storage node is not the storage node itself, it indicates that the third target storage node is not the storage node itself. The storage location of the file should be changed, so the third target file should be migrated to the new third target storage node.

In a third aspect, embodiments of the present disclosure provide a file management method for a client.

The method of the embodiment of the present disclosure is executed by the client, and the client can interact with the file storage system of the embodiment of the present disclosure, thereby using the file storage system for file management.

Figure 3 is a flow chart of a method for client file management provided by an embodiment of the disclosure. Referring to Figure 3, the method of the embodiment of the disclosure includes the following steps S301 to S303.

In step S301, the first target hash value corresponding to the first target file is determined according to the information of the first target file, and the first target storage corresponding to the first target hash value is determined according to the preset hash node correspondence relationship. node.

In some embodiments, the first target storage node is a storage node of a file storage system. The file storage system includes a management server and multiple storage nodes, and the hash node correspondence includes the corresponding hash of each storage node in the hash band. value. The hash band includes multiple hash values. The number of hash values corresponding to each storage node is proportional to the total capacity of each storage node. Each hash value in the hash band is assigned to each storage node. Each storage node corresponds to one or more hash values.

When the client needs to write a file (first target file) to the file storage system, it calculates the first hash value belonging to the hash band based on the information of the first target file (such as the file name), and obtains the first hash value from the management server. The hash node correspondence relationship determines the first target storage node corresponding to the first hash value, that is, the storage node where the first target file should theoretically be located.

In step S302, the final target storage node is determined according to the first target storage node.

In some embodiments, the proportion of used capacity in the first target storage node is negatively correlated with the probability that the first target storage node is determined to be the final target storage node.

After determining the first target storage node, continue to determine the final target storage node that actually stores the content of the first target file. The final target storage node may be the first target storage node, or it may be another storage node. The probability that the first target storage node is the final target storage node is negatively correlated with the proportion of the used capacity in the first target storage node (that is, the proportion of the used capacity in the first target storage node to its own total capacity).

It can be seen that the number of hash values corresponding to a storage node is proportional to its total capacity. Therefore, a storage node with a larger total capacity is more likely to be selected as the first target storage node and write the first target file to achieve balanced capacity allocation.

At the same time, after the storage node is selected as the first target storage node, the greater the proportion of used capacity (ie, the greater the load), the lower the probability that the first target file is actually written to it, thereby achieving load balancing. .

In step S303, a creation request for the first target file is sent to the final target storage node.

The creation request for the first target file is sent to the final target storage node, and the final target storage node executes the above file management method to actually write the first target file.

In some embodiments, determining the final target storage node based on the first target storage node includes generating a random value between 0 and 1.

When the random value is greater than the proportion of used capacity in the first target storage node, the first target storage node is determined to be the final target storage node.

When the random value is less than or equal to the proportion of used capacity in the first target storage node, other storage nodes other than the first target storage node are determined to be the final target storage nodes.

As a method of the embodiment of the present disclosure, a random value r between 0 and 1 can be calculated first, and then r is compared with the proportion w of the used capacity in the first target storage node. If r is greater than w, then The first target storage node is determined to be the final target storage node, and if r is not greater than w, other storage nodes are selected as the final target storage node.

In some embodiments, determining storage nodes other than the first target storage node as the final target storage node includes: determining the storage node with the smallest proportion of used capacity other than the first target storage node as the final target storage node.

As a method of the embodiment of the present disclosure, all storage nodes can be sorted in ascending order according to the proportion of used capacity w. When other storage nodes are to be selected as the final target storage nodes, the other storage nodes with the smallest w are selected as the final target. storage node.

It should be understood that the selection here can also be "iterative".

In some embodiments, the method of the embodiment of the present disclosure further includes: determining the corresponding second target hash value according to the information of the second target file, and determining the corresponding hash node according to the preset relationship. The second target storage node corresponding to the second target hash value is determined.

Send an operation request for the second target file to the second target storage node.

In some embodiments, the operations include opening the second target file, and/or deleting the second target file.

When the client wants to operate a certain file (such as a second target file) in the file storage system, it can calculate its second target hash value and find the second target storage corresponding to the second target hash value. The node sends a corresponding operation request to the second target storage node, so that the second target storage node performs the above file management method.

In this disclosed embodiment, the client can directly calculate the target storage node where the file is located, and the actual file management (such as storage) is also performed by each storage node according to certain rules instead of being uniformly performed by the file distribution server, so it It can ensure the balanced capacity of each storage node without causing excessive network load on a certain node, resulting in short file distribution delay and high efficiency of the entire system.

In the fourth aspect, referring to Figure 4, an embodiment of the present disclosure provides a management server, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor. Execution causes the processor to execute any file management method in the embodiments of the present disclosure.

In the fifth aspect, referring to Figure 5, an embodiment of the present disclosure provides a storage node, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor. Execution causes the processor to execute any file management method in the embodiments of the present disclosure.

In the sixth aspect, referring to FIG. 6 , an embodiment of the present disclosure provides a file storage system, which includes a management server and multiple storage nodes; the management server is the management server of the embodiment of the present disclosure; the storage node is the storage node of the embodiment of the present disclosure. .

In the seventh aspect, referring to Figure 7, an embodiment of the present disclosure provides a client, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor. Execution causes the processor to execute any file management method in the embodiments of the present disclosure.

In some embodiments, the processor is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory is a device with data storage capabilities, It includes but is not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I/O interface ( The read-write interface) is connected between the processor and the memory, and can realize the information interaction between the memory and the processor, including but not limited to the data bus (Bus), etc.

For example, the file storage system of the embodiment of the present disclosure can be used in a variety of different application scenarios. The detailed process of the file management method in each application scenario is introduced below.

(1) The process of a storage node accessing a file storage system (cluster) may include the following steps.

Referring to Figure 9, the management server (Management Center, MC) initializes the hash band according to the maximum number of storage nodes N supported by the system. The length of the hash band is 2 ^N and the interval distribution of the hash band is [0, 2 ^N -1].

For example, if the maximum number of storage nodes supported by the file storage system is 16, the hash band distribution is [0,65535].

When a storage node (Store Node, SN) accesses the file storage system, it first initiates an access request to the MC. The request content includes the total capacity of the storage node totalCap and the used capacity usedCap.

MC allocates storage node identifiers SNid based on the number of storage nodes on the network; the SNid distribution is [0, N-1].

For example, the SNid of the first storage node is 0, the SNid of the second storage node is 1, and the SNid of the third storage node is 2; after SNid1 is deleted, the newly connected storage node will be allocated the vacant SNid first, that is, in the fourth After a storage node is connected, the SNid is 2.

MC sends the allocated storage node ID SNid to the storage node, and the storage node saves its own SNid.

Referring to Figure 8, MC also analyzes the total capacity totalCap and used capacity usedCap in the storage node join request, and divides the hash band proportion of each storage node according to the total capacity of all existing storage nodes.

For example, if there are 3 storage nodes on the network, the total capacity of storage node 1 is 10000G, the total capacity of storage node 2 is 4000G, and the total capacity of storage node 3 is 6000G, then storage node 1 accounts for 50% of the total capacity, and storage node 2 accounts for 20%. , storage node 3 accounts for 30%, press up Rounding calculation shows that the hash band length (number of hash values) of storage node 1 is 2 ^N *0.5, that of storage node 1 is 2 ^N *0.2, and that of storage node 3 is 2 ^N *0.3.

Therefore, it is equivalent to establishing the corresponding relationship between the storage node identifier SNid and the hash value.

MC also calculates the weight factor of each storage node The weight factor w represents the proportion of used capacity in the total capacity.

After the new storage node is successfully connected or a storage node is deleted, MC immediately synchronizes the latest hash strip information HashStripe and storage node information SnInfoList to all storage nodes in the file storage system.

In some embodiments, HashStripe contains the hash distribution of each storage node (hash node correspondence), and SnInfoList contains all storage node information in the file storage system, including obtaining the IP, port and storage node of the storage node by SNid index. weight w.

The client can synchronize HashStripe and SnInfoList information from MC regularly.

(2) The process of creating (writing) a file may include the following steps.

Referring to Figure 11, the client uses the file name fileName to be created to hash and calculate its hash value hashid=hash(fileName); and calculates hashid mod 2 ^N to determine the SNid of the selected storage node (the first target storage node).

The client parses the IP, port and storage node weight w of the storage node from the SnInfoList based on the SNid.

The client generates a random factor (random value) in the following way: That is, r belongs to the interval of (0, 1); and compare the weight w with r. If r ≤ w, it means that the weight does not meet the requirements, and iteratively selects the next storage node as the final target storage node in ascending order of w; if r>w, it means that the weight meets the requirements, then the storage node is selected as the final target storage node.

In order to maintain a balanced capacity in the entire file storage system, files should be created first on storage nodes with small used capacity. Since w represents the proportion of used capacity, for storage nodes with the same total capacity, the smaller the used capacity, the smaller w, and the smaller the interval smaller than w in the (0,1) interval, on the contrary r> The greater the probability of w, the greater the probability that the storage node will be selected, thereby balancing the storage node capacity in the file storage system.

After the client selects the final target storage node, it constructs a file creation request and sends it to the final target storage node (which may be the first target storage node). The request content includes the file name fileName and the file capacity fileCap.

Referring to Figure 12, after receiving the file creation request, the final target storage node first parses out fileName and SNid ₁ .

Calculate hashid=hash(fileName), and determine whether the file already exists locally based on the file name fileName. If it already exists, respond directly to the creation failure.

If the file does not exist locally, continue to check whether the file link exists locally (refer to Figure 10 for the link structure); if a file link exists, parse the file link information to obtain the IP and port of the target storage node (the storage node pointed by the link), and Initiate a query request to the target storage node to check whether the file is valid; if the file is not in the target storage node pointed by the link, it means that the link is invalid and the file can be created; if the file is still in the target storage node pointed by the link, it means that the file storage system has been The file exists and the direct response to the client fails.

If the file does not exist and the link does not exist (or the link is invalid), the creation of the file begins.

Compare the SNid ₁ obtained from the HashStripe with its own storage node ID SNid ₂ to see if they are equal (that is, determine whether it is the first target storage node). If they are equal, it means that the file should be created in this storage node. Create the file and respond to the success of the creation. client.

If SNid ₁ and SNid ₂ are not equal, it means that the file should have been created on storage node SNid _1. At this time, a file link should be created on SNid ₁ , which points to SNid ₂ , and the real file still exists on SNid ₂ . . Therefore, SNid ₂ initiates a file link creation request to SNid _1. The request content includes the target storage node identifier SNid ₂ and the file name.

After SNid ₁ receives the request to create a file link, it checks whether the file or file link already exists on this storage node according to the previous method (and checks whether the file link is valid); if one of them exists, it responds to SNid ₂ that the file already exists; If neither the file nor the file link exists, the file link is created, indexed by hashid, and the link is stored locally.

In the above process, if SNid ₁ returns that the file already exists, it directly responds to the client that the file creation failed; if SNid ₁ returns that the file creation link is successful, SNid ₂ creates the file fileName and responds to the client.

After the file is successfully created, the used capacity of the storage node is updated to usedCap+fileCap and reported to MC.

(3) The process of file reading and writing (opening) may include the following steps.

Referring to Figure 13, the client performs hash calculation hashid=hash(FileName) on the file name of the file to be read and written, obtains the storage node identifier SNid from the hash band according to the hashid, parses the corresponding storage node information from SnInfoList, and then Initiate a file open request to the storage node, and the request content includes the file name FileName.

After receiving the request, the storage node first checks whether the local file exists based on the file name. If it exists, it directly opens the file and responds to the client with the file handle.

If the file does not exist, hash calculation is performed based on the file name hashid=hash(FileName), and then the file link list (FileLinkList) is queried based on the hashid to determine whether a link to the file exists. If it does not exist, the file is directly responded to the client. There are no errors.

If the file link exists, index the information of the target storage node (the final target storage node that actually stores the file content) according to the hashid, respond to the target storage node information to the client, and set the response type to file link.

After the client receives the response, if the response type is a file link, it parses out the target storage node pointed by the link and initiates a file open request to the storage node.

After the client obtains the handle, it sends a read and write request to the target storage node. The request content includes the file handle, read and write length, and read and write offset.

(4) The process of deleting files may include the following steps.

The client calculates the hashid based on the file name and selects the storage node SNid ₁ based on the hash band, and initiates a file deletion request to the storage node. The request content includes the file name fileName.

After storage node SNid ₁ receives the deletion request, it first checks whether the file exists, and if it exists, deletes the file directly.

If the file does not exist, continue to detect whether the file link exists. If the file link exists, obtain the link information based on the hashid, and respond to the client with the SNid ₂ information of the target storage node pointed to by the link. The response type is set to file link. If the link If it does not exist, an error will be reported.

If the response type is a file link, the client parses the target storage node in the response. IP and port information of SNid ₂ , and initiate a file deletion request to SNid ₂ .

If SNid ₂ responds successfully to deleting the file, the client initiates a deletion file link request to SNid ₁ .

(5) The process of file link management may include the following steps.

The File Link Management module of each storage node is responsible for cleaning up invalid file links of this storage node.

The file link management module periodically traverses the FileLinkList to obtain the link information, and then initiates a query to the target storage node to see if the file is valid. If the file has been deleted, it means that the link is invalid, and the file link management module will directly remove the link.

(6) The process of hash band management may include the following steps.

The hash stripe management module (Hash Stripe Management) of the storage node is responsible for file migration when the hash strip is updated.

After the storage node updates the hash band, the hash band management module traverses the file names that have been created by the storage node, then hashes and calculates hashid=hash(fileName), and then takes out SNid ₁ and SNid from the new and old hash bands respectively. _2. If the SNids are the same, no processing will be performed. If they are not the same, it means that the file should be migrated to SNid ₁ .

SNid ₂ first initiates a file link request to SNid ₁ , and SNid ₁ creates a file link to the file after receiving the request.

Those of ordinary skill in the art can understand that all or some of the steps, systems, and functional modules/units in the devices disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof.

In hardware implementations, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may consist of several physical components. Components execute cooperatively.

Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, Such as application specific integrated circuits. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage medium includes all media used to store information, such as volatile and non-volatile, removable and non-removable media implemented in any method or technology (computer readable instructions, data structures, program modules or other data). Computer storage media includes but is not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk memory ; Compact Disk Read-Only (CD-ROM), Digital Versatile Disk (DVD) or other optical disk storage; Magnetic cassette, magnetic tape, disk storage or other magnetic storage; Any other storage device that can be used to store desired information and can be accessed by a computer medium. Additionally, it is known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Example embodiments have been disclosed, and although specific terms are employed, they are used and should be interpreted in a general illustrative sense only and not for purpose of limitation. In some instances, it will be apparent to those skilled in the art that, unless expressly stated otherwise, features, characteristics, and/or elements described in connection with a particular embodiment may be used alone, or may be combined with features, characteristics, and/or elements described in connection with other embodiments. and/or used in combination with components. Accordingly, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

A method of file management, used for a management server of a file storage system. The file storage system includes a management server and multiple storage nodes. The method includes:

Determine a hash band that includes multiple hash values;

The hash node correspondence is determined according to the hash band and the total capacity of each storage node in the file storage system; the hash node correspondence includes the hash corresponding to each storage node in the hash band. Hash values, the number of hash values corresponding to each storage node is proportional to the total capacity of each storage node, and each hash value in the hash band is allocated to each storage node; and

Send the hash node correspondence to each storage node.
The method of claim 1, wherein,

The total number of hash values in the hash band is determined based on the maximum number of storage nodes that the file storage system can support.
The method of claim 1, further comprising:

When a storage node is changed in the file storage system, the updated hash node correspondence is determined based on the hash band and the total capacity of each storage node in the file storage system after the change; the storage node change includes: Add new storage nodes, and/or delete existing storage nodes; and

Send the updated hash node correspondence to each storage node.
A method of file management, used for the first storage node of a file storage system. The file storage system includes a management server and multiple storage nodes. The method includes:

Receive a creation request for the first target file;

Determine the first target hash value corresponding to the first target file according to the information of the first target file, and determine the first target storage node corresponding to the first target hash value according to the preset hash node correspondence relationship. ; The hash node correspondence includes the hash value corresponding to each storage node in the hash band, the hash band includes multiple hash values, each storage node The number of hash values corresponding to a node is proportional to the total capacity of each storage node, and each hash value in the hash band is allocated to each storage node;

The first storage node is the first target storage node, and the first storage node locally stores the first target file; and

If the first storage node is different from the first target storage node, then the first storage node locally stores the first target file and creates a copy of the first target file in the first target storage node. Link.
The method of claim 4, further comprising:

Receive an operation request for the second target file; the operation includes opening the second target file, and/or deleting the second target file;

If the second target file is stored locally, the second target file is operated according to the operation request; and

If the second target file is not stored locally, the operation request is responded to based on the link of the second target file.
The method of claim 4, further comprising:

Receive the updated hash node correspondence;

The third target hash value corresponding to the third target file is determined based on the information of the third target file stored locally on the first storage node, and the third target hash is determined based on the updated correspondence between the hash nodes. The third target storage node corresponding to the value; and

If the first storage node is different from the third target storage node, the third target file is migrated to the third target storage node.
A file management method for a client, the method includes:

Determine the first target hash value corresponding to the first target file according to the information of the first target file, and determine the first target storage node corresponding to the first target hash value according to the preset hash node correspondence; The first target storage node is a storage node of a file storage system. The file storage system includes a management server and multiple storage nodes. The hash node correspondence includes the hash value corresponding to each storage node in the hash band. , as mentioned The hash band includes multiple hash values, the number of hash values corresponding to each storage node is proportional to the total capacity of each storage node, and each hash value in the hash band is allocated to each storage node;

Determine the final target storage node according to the first target storage node; the proportion of used capacity in the first target storage node is negatively correlated with the probability that the first target storage node is determined to be the final target storage node; and

Send a creation request for the first target file to the final target storage node.
The method according to claim 7, wherein determining the final target storage node according to the first target storage node includes:

Generate a random value between 0 and 1;

If the random value is greater than the proportion of used capacity in the first target storage node, then the first target storage node is determined to be the final target storage node;

If the random value is less than or equal to the proportion of used capacity in the first target storage node, then the other storage nodes other than the first target storage node are determined to be the final target storage nodes.
The method according to claim 8, wherein determining that other storage nodes other than the first target storage node are the final target storage nodes includes:

The storage node with the smallest proportion of used capacity other than the first target storage node is determined to be the final target storage node.
The method of claim 7, further comprising:

Determine the second target hash value corresponding to the second target file according to the information of the second target file, and determine the second target storage node corresponding to the second target hash value according to the preset hash node correspondence; and

Send an operation request for the second target file to the second target storage node; the operation includes opening the second target file, and/or deleting the second target file.
A management server including one or more memories, one or A plurality of processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor, so that the processor executes the file management method according to any one of claims 1 to 3 .
A storage node, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor, so that the processor Perform the file management method according to any one of claims 4 to 6.
A file storage system, which includes a management server and multiple storage nodes;

The management server is the management server of claim 11; and

The storage node is the storage node according to claim 12.
A client, which includes one or more memories and one or more processors; the memory stores a computer program that can be executed by the processor, and the computer program is executed by the processor, causing the processor to execute the right The method of file management according to any one of claims 7 to 10.