JP6648596B2 - File system control device, storage system, file system control method, and program - Google Patents

Description

  The present invention relates to a file system control device, a storage system, a file system control method, and a program therefor.

  The CAS (Content Addressed Storage) stores data using a content address (hereinafter referred to as CA) determined based on the content of the data to be stored (for example, a hash value of the data).

  For this reason, the CAS does not need to separately store data of the same content, eliminates duplicate recording, and can reduce the data capacity.

  Patent Literature 1 discloses a technology related to a content address type storage system that specifies a storage location where data is stored by a unique address specified according to the content of the stored data.

  Patent Document 2 discloses a technique in which actual data is not stored, but instead metadata is used to track the position of all data in the system, and a large number of users access the latest file in synchronization. I have.

  Patent Literature 3 discloses a technique related to a storage system in which an indirect address is permanently allocated in a file system structure to prevent a CA recalculation from being propagated to a file system root even when a block is rewritten. I have.

  Patent Literature 4 discloses a technology related to a storage system in which the time and load required for data copy processing in a CAS are suppressed, and a decrease in system performance is suppressed.

WO2012 / 109833 JP-T-2015-512071 Japanese Patent No. 5556025 JP 2010-198276 A

  In a device that stores a file system structure in a tree structure on a CAS, there is a performance problem when updating a file.

  A normal storage stores data by using a data storage position as an address. In contrast, the CAS stores data using a CA determined based on the content of the data to be stored. That is, the CAS determines the CA for reference after storing the data. For this reason, the CAS needs to wait for the completion of storing the lower block in order to determine the content of the block of the upper tree, and it takes time to store the tree structure. This operation is a performance issue especially in a system with a large latency.

  A performance problem at the time of updating a CAS file will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a file update operation in CAS.

  FIG. 1 is a diagram for explaining an example of updating “data2”, which is one of the data blocks configuring “/ dir1 / file1”. The upper file system structure 1000 in FIG. 1 shows an original block group, and the lower file system structure 2000 shows a block group in a file update process. The file system structure 1000 is a tree structure. The left block in FIG. 1 is the upstream side, and the right block is the downstream side. “Ca00” to “ca61” are CAs (content addresses).

  For example, in the block of CA “ca20”, CA “ca40” which is block information of “dir1” is stored. In the block of CA “ca50”, CA “ca60”, “ca61” and “ca62” which are block information of “file1” are stored. Data is stored in the block “data2” of the CA “ca61”. A block in which data is stored is called a data block. The CAS performs a file update that updates the block "data2" as follows.

  First, the CAS duplicates “data2”, modifies the duplicated block “data4” (data in which the hatched portion in (1) in the figure is changed), and calculates a new CA “ca63”. ((2) in the figure). In CAS, deduplication is realized by integrating blocks of the same content into one block and referring to the blocks from a plurality of upper blocks. When a change is made to an existing block, the existing block must be duplicated without fail.

  Next, the CAS first copies the block list of “file1” of CA “ca50” in order to reflect the CA “ca63” of the changed block in “file1”. Thereafter, the CAS updates (replaces) the CA “ca61” included in the duplicated block list with “ca63”, generates “ca51” by recalculating the CA, and updates “ca50” to “ca51”. (Replace) ((3) in the figure). As described above, the CAS needs to sequentially perform the duplication and the CA update / recalculation for the upper block of the tree, and must sequentially perform the update until the root (root) located at the top of the tree is reached. No ((4) in the figure).

  In particular, in a CAS having a large latency, the following operation is a major performance problem.

  First, the first problem is that the CA recalculation processing at the time of file update propagates over a wide range (the above-described operations (3) and (4)). In CAS, a value calculated from the contents of a block (for example, a hash value of data) is used as CA. Therefore, in CAS, when a data block of a file is rewritten, the CA of the data block is changed. Then, the CA of the file storing the block information is changed, the CA of the directory storing the file information is changed, and so on, and the recalculation of the CA is propagated until the root of the file system is reached. I do. In particular, when files are updated frequently, it becomes inefficient.

  The second problem is that a copy is required when updating a file (the above-described operation (1)). In CAS, since blocks having the same content have the same CA, one block may be referred to by a plurality of upper blocks. For this reason, when rewriting a block, it is necessary to copy the original block first, make a change to the copied block, and store it, instead of overwriting the original block as it is. Due to this duplication process, the response performance at the time of updating the file becomes slow.

  Patent Literature 1 discloses a general operation of a CAS, in which CA recalculation processing at the time of file update propagates over a wide range, and duplication is required at the time of file update.

  Patent Document 2 does not mention improvement in performance at the time of file update in CAS.

  Patent Document 3 can prevent the CA recalculation from propagating to the root of the file system, but does not mention improvement in performance at the time of file update.

  Patent Literature 4 only describes the load reduction of the copy processing in the CAS. The CA recalculation processing at the time of file update propagates over a wide range, and the file update requires copying.

  Therefore, none of the techniques described in the above-mentioned documents have solved the problem that the CA recalculation processing at the time of file update propagates in a wide range in the CAS, and that duplication is required at the time of file update. There is a problem with the performance at the time.

  Therefore, an object of the present invention is to solve the above-mentioned problem. In the CAS, the CA recalculation processing at the time of file update propagates over a wide range, and duplication is required at the time of file update. An object of the present invention is to provide a file system control device or the like that solves the problem of the existence.

  The file system control device of the present invention updates data, and calculates a first content address of the data block based on a file path and an offset of the data block at a location where the data is updated. A file system reproducing unit that calculates the first content address of an upper block storing the data block updated by the data reproducing unit; and a file system determination that converts the first content address into a second content address. Means.

  The file system control method of the present invention updates data, calculates a first content address of the data block based on a file path of an updated location of the data and an offset of the data block, and updates the updated data block. The first content address of the upper block to be stored is calculated, and the first content address is converted to a second content address.

  The computer program of the present invention updates data, calculates a first content address of the data block based on a file path and an offset of the data block at an updated location of the data, and updates the updated data block. The program causes the computer to execute a process of calculating the first content address of the upper block to be stored and a process of converting the first content address into a second content address.

  According to the present invention, in the CAS, it is not necessary to propagate the CA recalculation processing at the time of updating a file over a wide range, and duplication is not required at the time of updating the file, so that there is an effect that the performance at the time of updating the file is not reduced.

FIG. 1 is a diagram illustrating an example of a file update operation in CAS. FIG. 2 is a block diagram illustrating an example of a configuration of the storage system according to the first embodiment. FIG. 3 is a diagram illustrating an example of an initial state in which directories, files, data blocks, and their tree structures are stored in a data storage unit. FIG. 4 is a diagram illustrating an example of an operation in which the file system control device updates “file1” in the state of FIG. 3. FIG. 5 is a diagram illustrating an example of an operation of determining data by the file system determination unit. FIG. 6 is a block diagram illustrating an example of a configuration of a file system control device according to the second embodiment.

<First embodiment>
A first embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 2 is a block diagram illustrating an example of a configuration of the storage system 10 according to the first embodiment.

  The storage system 10 includes a file system control device 100 and a data storage unit 200.

  The file system control device 100 includes a file system access unit 101, a file system reproduction unit 102, a data reproduction unit 103, and a file system determination unit 104.

  The file system access unit 101 provides a means for accessing a file system (data storage unit 200) from a client. File system access can be classified into two types: operations relating to a file system structure such as open, close, mkdir, and unlink, and operations relating to data such as read and write. Since the file system structure in the present embodiment is a tree structure, the file system structure is hereinafter also referred to as a tree structure. The file system access unit 101 distributes an operation request relating to the former operation to the file system reproduction unit 102 and a data transfer request relating to the latter operation to the data reproduction unit 103.

  The file system reproduction unit 102 receives a tree structure operation request from the file system access unit 101, reads information (directory, file, data block) necessary for the tree structure operation from the data storage unit 200, and holds the result. . The information held in the file system reproduction unit 102 is called first information.

  The data reproduction unit 103 receives a data transfer request from the file system access unit 101, and performs read and write processing on the data storage unit 200. For example, when performing a write process, the data reproduction unit 103 holds the write data. The information held in the data reproduction unit 103 is called second information.

  The file system determination unit 104 detects a file that has not been updated for a certain period of time, and stores the first information and the second information (the tree structure and the write data) held by each of the file system reproduction unit 102 and the data reproduction unit 103. ) Is stored in the data storage unit 200. In addition, the file system determination unit 104 can also instruct the file system reproduction unit 102 and the data reproduction unit 103 to write out all the information held by each to the data storage unit 200.

  Here, the file system access unit 101, the file system reproduction unit 102, the data reproduction unit 103, and the file system determination unit 104 are configured by a hardware circuit such as a logic circuit.

  In the file system access unit 101, the file system reproduction unit 102, the data reproduction unit 103, and the file system determination unit 104, the processor of the file system control device 100, which is a computer, executes a program on a memory (not shown). It may be a functional unit realized by this.

  The data storage unit 200 is configured by a storage device such as a disk device and a semiconductor memory.

  Note that the file system reproduction unit 102 and the data reproduction unit 103 also have a memory function of holding data inside.

  A processing image for updating a file stored in the data storage unit 200 will be described below with reference to FIGS. 3 to 5 each illustrate the data storage unit 200, the file system reproduction unit 102, and the data reproduction unit 103 together with the blocks or block groups stored therein for the sake of explanation.

  FIG. 3 is a diagram illustrating an example of an initial state in which a tree structure including directories, files, and data blocks is stored in the data storage unit 200.

  Note that the tree structure in FIG. 3 is an original file system structure, and is the same as that in FIG. 1 described above, and thus detailed description is omitted.

  FIG. 4 is a diagram illustrating an example of an operation in which the file system control device 100 updates the block information of “file1” in the state of FIG. The following (1) to (4) show specific procedures of the operation of the file system control device 100.

  (1) First, the file system access unit 101 notifies the file system reproduction unit 102 of a write (update) request for “file1” received from the client.

  (2) Next, the file system reproduction unit 102 reads the information of the tree structure of “file1” from the data storage unit 200, and returns the CA “ca61” of the data block corresponding to the update location to the file system access unit 101. . Here, the file system reproduction unit 102 holds the information of the tree structure of “file1”.

  (3) Then, the file system access unit 101 notifies the data reproduction unit 103 of the write request from the client and the CA “ca61” of the original block. The data reproduction unit 103 writes the update data in the new block “data2 ′” without duplicating the original block “data2”, and holds the offset and size of the update data and the CA “ca61” of the original block. Then, the data reproduction unit 103 calculates the CA of the block storing the update data by a new method described later, and returns “ca61 ′” to the file system access unit 101.

  Note that the above item (3) corresponds to the description in the frame of the data reproduction unit 103 in FIG. For example, the new block corresponds to the portion described as "data2 '". Further, for example, CA “ca61” of the original block corresponds to “Original data”, and CA “ca61 ′” of the new block corresponds to “Targeted data”. Then, the offset and size of the update data correspond to “Modified”. The specific values of the offset and size of the update data are, for example, “129: 85” and “257: 42” as shown in FIG. Corresponding. The unit of the offset and the size is, for example, a bit or a word.

  The new method is, for example, a function for calculating a CA from a file ID (IDentification) (or full path) and an offset of a data block. The offset of the data block is, for example, the position information of “ca61 ′” with respect to “file1”. In the calculation using the new method, when the file ID is “0100” and the offset of the data block is “0010”, the CA of “ca61 ′” is set to “0110”, for example, in binary notation.

  (4) Thereafter, the file system access unit 101 notifies the received CA “ca61 ′” to the file system reproduction unit 102. Then, the file system reproduction unit 102 recalculates “file1”, which is the file storing the information of the rewritten block, and the higher-order CA to the root of the file system by the new method. The recalculated CAs are "ca50 '", "ca40'", "ca20 '", "ca10'", and "ca00 '", as shown in FIG.

  As described in (3) above, when updating the file, the file system control device 100 writes only the changed part, and the part that has not been changed is referred to the original block. As a result, the file system control device 100 can minimize the writing amount and shorten the response time.

  Further, in the above (3) and (4), the file system control apparatus 100 calculates the CA by the new method, so that when updating the same file a plurality of times, the recalculation of the CA is performed only for the first time and the second time. Subsequent updating eliminates the need to recalculate CA. In order to prevent CA collision between the existing system and the new system, for example, the leading bit of the CA is set to “0” in the existing system. Also, for example, in the new method, the first bit is “1”. In the case of the new system, using the above example, for example, “1” is added to the first bit of “0110”, and the CA becomes “10110”. Note that the CA of the new method is also referred to as a first content address. The CA of the existing system is also called a second content address.

  FIG. 5 is a diagram illustrating an example of an operation in which the file system determination unit 104 determines data.

  First, the file system determination unit 104 detects a file that has not been updated for a certain period of time, and instructs the file system reproduction unit 102 and the data reproduction unit 103 to write the corresponding information held by each to the data storage unit 200. . At this time, the data reproduction unit 103 duplicates the unchanged portion referred to as the original block, and then writes it out to the data storage unit 200. The above-mentioned predetermined period is set in advance by the file system administrator based on the CAS latency or the like.

  Next, the file system determination unit 104 recalculates the CA by the existing method based on the content of the block received by the data storage unit 200 via the file system reproduction unit 102 and the data reproduction unit 103, and Update the file system structure of Note that, as described above, the existing method is an address determined based on the content of the stored data. When all of the lower blocks become CAs of the existing method, the file system determination unit 104 recalculates the CA of the upper block by the existing method.

  Incidentally, the above is an example of storing a file system having a hierarchical directory structure on a CAS. However, even when a file system having no hierarchy, such as an object storage, is stored on the CAS, improvement in performance at the time of updating a file can be expected by applying this embodiment.

  As described above, the storage system 10 of the present embodiment introduces a new CA calculation method that does not depend on the binary representation (data content) of data and mixes it with the existing method. The storage system 10 normally uses an existing method, but uses a new method when updating a file.

  The existing method calculates CA from the contents of a block. If the contents of the blocks are the same, they have the same CA, so that the storage system 10 can enjoy the effect of deduplication.

  On the other hand, the new method calculates CA from the file ID (or full path) and the offset of the data block. The new method uses a unique CA in the system and eliminates the need for duplication processing when updating a file. However, in the new method, since the CA is unique, there is no deduplication effect.

  As described above, first, in the new method, since the CA does not depend on the content of the block, the CA recalculation when the data block is rewritten becomes unnecessary, and the upper block does not need to be rewritten. However, the new method cannot enjoy the effect of deduplication. To solve this, the storage system 10 introduces a mechanism for asynchronously (post-process) rewriting the CA calculated by the new method to the CA of the existing method. Thereby, the storage system 10 can enjoy the effect of deduplication in the long term while improving the efficiency of the file update process.

  Secondly, in the new method, since the CA is unique and one block is not referred to by a plurality of blocks, it is not necessary to duplicate the original block when updating the file, and the block can be rewritten as it is. Thereby, the storage system 10 can shorten the response time at the time of updating the file. However, when the block is rewritten for the first time, since the CA of the original block is of the existing system, the block must be copied. To solve this, the storage system 10 stores only the update data in the new block, stores the CA of the original block and information (data offset value, data size) of the data to be rewritten, and there is no change. The location is a reference to the original block and is not duplicated. Thereby, the storage system 10 can also shorten the response performance at the time of the first block rewriting. In addition, the storage system 10 duplicates the location referred to by a post process.

  As described above, the storage system 10 according to the present embodiment allows a plurality of CA systems to coexist on the CAS storing the file system, and determines the transition between the CA systems to data stability (for long-term updating). No, etc.).

  In addition, the storage system 10 can improve the file update performance without decreasing the duplication rate over a long period of time in a CAS with a large latency.

  Further, the storage system 10 is a system that frequently updates the same file, and can improve the performance when updating the file without reducing the duplication rate.

  As described above, the storage system 10 optimizes the CA recalculation processing and performs the post-processing for the copy processing when updating the file.

  The storage system 10 according to the present embodiment has the following effects.

  That is, in the CAS, the problem that the CA recalculation processing at the time of updating a file propagates over a wide area and that the duplication is required at the time of updating the file, and that there is a performance problem at the time of updating the file, is achieved. .

The reason is that the CA is calculated by the new method from the file path and the offset of the data block, and the CA calculated by the new method is asynchronously rewritten to the CA of the existing method.
<Second embodiment>
Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 6 is a block diagram illustrating an example of a configuration of the file system control device 300 according to the second embodiment. The second embodiment corresponds to an example of the minimum configuration of the file system control device 300 of the first embodiment.

  The file system control device 300 includes a data reproduction unit 301, a file system reproduction unit 302, and a file system determination unit 303.

  The data reproducing unit 301 updates the data, and calculates a first content address of the data block based on the file path of the updated location of the data and the offset of the data block.

  The file system reproduction unit 302 calculates a first content address of an upper block that stores the data block updated by the data reproduction unit.

  The file system determination unit 303 converts the first content address into a second content address.

  The file system control device 300 according to the present embodiment has the following effects.

  That is, in the CAS, the problem that the CA recalculation processing at the time of updating a file propagates over a wide area and that the duplication is required at the time of updating the file, and that there is a performance problem at the time of updating the file, is achieved. .

  The reason is that the data is updated, the first content address of the data block is calculated based on the file path of the data update location and the offset of the data block, and the first content address of the upper block that stores the updated data block is calculated. Calculate the content address. Then, the first content address is converted into the second content address.

  The embodiment of the present invention has been described with reference to the drawings, but the present invention is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Storage system 100 File system control device 101 File system access part 102 File system reproduction part 103 Data reproduction part 104 File system determination part 200 Data storage part 300 File system control device 301 Data reproduction part 302 File system reproduction part 303 File system determination part 1000 File system structure 2000 File system structure

Claims (10)

Updating data, calculating a first content address of the data block based on a file path and an offset of the data block at a location where the data is updated, data reproduction means,
Calculating a first content address of an upper block storing the data block updated by the data reproducing unit, a file system reproducing unit,
A file system determination unit that converts the first content address into a second content address.   The file system control device according to claim 1, wherein the second content address is determined based on the content of the data.   3. The file according to claim 1, wherein the second content address sets a first bit of the content address to “0”, and the first content address sets a first bit of the content address to “1”. 4. System control unit. The file system control device according to any one of claims 1 to 3,
A data storage means for storing the data. Updating data, calculating a first content address of the data block based on a file path and an offset of the data block at a location where the data is updated,
Calculating the first content address of the upper block storing the updated data block;
A file system control method for converting the first content address into a second content address.   6. The file system control method according to claim 5, wherein the second content address is determined based on the content of the data.   7. The file according to claim 5, wherein the second content address sets the first bit of the content address to “0”, and the first content address sets the first bit of the content address to “1”. 8. System control method. Updating data, and calculating a first content address of the data block based on a file path and an offset of the data block at an updated location of the data;
A process of calculating the first content address of an upper block that stores the updated data block;
Converting the first content address into a second content address.   The program according to claim 8, wherein the second content address is determined based on a content of the data.   The program according to claim 8, wherein the second content address sets the first bit of the content address to “0”, and the first content address sets the first bit of the content address to “1”. .

Images