JP6330824B2 - Storage system, access device, client device, method and program - Google Patents

Description

  The present invention relates to a storage system having a function of storing data with the same content eliminated.

  In recent years, storage systems that store data with the same content eliminated are known. Hereinafter, eliminating duplicate data having the same content is also referred to as deduplication. In particular, there is a storage system that divides data to be written into a plurality of data blocks and eliminates duplication of data blocks having the same contents even if the data blocks constitute different data. Such a storage system stores data blocks that have not yet been stored among data blocks obtained by dividing data to be written. Such a storage system stores, for data to be written, a reference to a data block constituting the data.

  For example, Patent Literature 1 describes an example of a technique for performing deduplication by dividing data blocks in real time while receiving data to be written. In this related technique, data input to the data buffer is divided according to a predetermined standard. In this related technique, data that remains in the data buffer without being divided and data that is continuously input are connected and divided according to a predetermined criterion.

  Further, in a storage system having such a deduplication function, one data block can be referred to by a plurality of data, and thus high reliability corresponding to the loss of the data block is required. Therefore, in such a storage system, data blocks are distributed and arranged with redundancy. Specifically, the storage system further divides the data block into fragments and adds a parity according to a predetermined redundancy. Then, the storage system stores the fragment and the parity in a plurality of storage devices.

  For example, Patent Document 2 describes an example of a technique for setting the redundancy of a data block according to the number of data that refers to the data block. The number of data referring to a certain data block is hereinafter referred to as the reference number. Specifically, in this related technique, an appropriate redundancy is set in advance for the number of references. And this related technique detects the difference between the appropriate redundancy determined with respect to the number of references and the actual redundancy for each stored data block for each predetermined timing. Rewrite the data block with a high degree of redundancy.

  Further, for example, Patent Document 3 describes an example of a technique corresponding to writing of a data block with different redundancy even if the content is the same as that of an existing data block. In this related technique, when the redundancy required for the data block to be written is higher than the redundancy of the stored data block having the same content, the data block is newly stored with the new redundancy. In addition, this related technique deletes data blocks with lower redundancy while leaving data blocks with higher redundancy among data blocks with the same content at a predetermined timing. In this related technique, the reference from the data referring to the deleted data block is changed to the reference to the remaining data block.

JP 2014-174604 A JP 2010-224845 A JP 2011-159142 A

  However, the related art described above has the following problems.

  In a storage system having a deduplication function as described above, the desired redundancy varies depending on the number of nodes constituting the storage. For example, when the number of nodes is 2, in order to enable access to stored data even when a failure occurs in one node, 50% of the entire data needs to be parity. When the number of nodes is 4, 25% of the entire data needs to be parity.

  Here, in the storage system, since the number of nodes generally increases or decreases due to addition of a node due to lack of capacity or replacement of old hardware with new hardware, there is a possibility that desirable redundancy may change. Therefore, in such a storage system, there is a case where it is desired to change the redundancy of stored data.

  However, Patent Document 1 does not describe the redundancy of data blocks stored by performing deduplication.

  In addition, as described above, the related technique described in Patent Document 2 rewrites data blocks with appropriate redundancy for each data block at a predetermined timing when the data blocks are not appropriately redundant. In order to rewrite data with an appropriate redundancy, it is necessary to read the existing data block. Therefore, when this related technology is applied when it is desired to change the redundancy in the entire storage system, a redundancy conversion process is performed in which all data blocks are read and rewritten with a new redundancy. . For this reason, when the amount of stored data is large, a situation in which the load for a long time is high is caused by batch conversion processing of redundancy for all data blocks. In such a situation, in order to suppress the influence on the performance of accessing data, it is assumed that access in operation is prioritized and the speed of the redundancy conversion process is suppressed. In that case, it takes a long time to complete the redundancy conversion process for all the data blocks, and the efficiency of the conversion process is poor.

  In addition, as described above, the related art described in Patent Document 3 has a redundancy required for a data block to be written higher than that of a data block stored with the same content. A new data block is stored with the required redundancy. Therefore, if this related technique is applied when it is desired to change the redundancy in the entire storage system, the redundancy is sequentially converted when data blocks are written. Therefore, the process of reading the data block before conversion is not required for the redundancy conversion process. Further, no batch redundancy conversion processing for all data blocks occurs. For this reason, this related technique can change the redundancy more efficiently than the related technique described in Patent Document 2. However, according to this related technique, when data is written after a change in redundancy is requested, writing with the required redundancy is required even if a data block having the same contents is already stored. That is, in this related technique, after a change in redundancy is requested, writing is performed for any data block of any data. As a result, this related technique degrades the write performance that should be improved by deduplication compared to the normal operation due to the change in redundancy.

  The present invention has been made to solve the above-described problems. That is, an object of the present invention is to provide a technique for efficiently changing redundancy while suppressing the influence on performance in a storage system having a deduplication function.

  The storage system according to the present invention includes a distributed storage unit that distributes and stores redundantly processed data in a plurality of storage devices, and divides the write data requested to be written into data blocks. A data block in which a data block having the same content is not stored in the distributed storage unit and a data block in which a data block having the same content is already stored in the distributed storage unit and the redundancy does not satisfy the required value The data block extracted from the data block selected as a write target, and the data block selected by the selection unit is subjected to redundancy processing based on the redundancy satisfying the required value and is stored in the distributed storage unit Of the data blocks stored in the writing unit and the distributed storage unit, the redundancy satisfies the required value. A conversion unit that converts a data block that has not been converted into a data block that has been subjected to redundancy processing based on redundancy satisfying the required value, and that is stored in the distributed storage unit while suppressing processing speed; Is provided.

  The access device according to the present invention includes the selection unit, the writing unit, and the conversion unit in the storage system.

  Another access device of the present invention includes the write unit and the conversion unit in the storage system.

  The client device of the present invention includes the selection unit in the storage system.

  Further, according to the method of the present invention, the computer device divides the write data requested to be written into data blocks by using a distributed storage unit that stores the redundantly processed data in a plurality of storage devices. Among the divided data blocks, a data block having the same content data block not stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit and the redundancy of the data block is the required value. Select the data block extracted from the data blocks that do not satisfy as a target for writing, and for the selected data block, perform redundancy processing based on the redundancy satisfying the required value, and store in the distributed storage unit, Among data blocks stored in the distributed storage unit, data blocks whose redundancy does not satisfy the required value It is converted into the data block to which the redundancy processing based on the redundancy satisfying the request value, a conversion process to be stored in the distributed storage unit, executes while suppressing the processing speed.

  In addition, the program of the present invention uses a distributed storage unit that distributes and stores redundantly processed data in a plurality of storage devices, divides the write data requested for writing into data blocks, and generates divided data. Among the blocks, a data block in which a data block having the same content is not stored in the distributed storage unit, and a data in which a data block having the same content is stored in the distributed storage unit and the redundancy does not satisfy the required value The computer apparatus is caused to execute a selection step of selecting a data block extracted from the blocks as a write target.

  The present invention can provide a technology for efficiently changing redundancy while suppressing the influence on performance in a storage system having a deduplication function.

1 is a block diagram showing a configuration of a storage system as a first embodiment of the present invention. FIG. It is a figure which shows an example of the hardware constitutions of the storage system as the 1st Embodiment of this invention. It is a schematic diagram explaining the data block and fragment in the 1st Embodiment of this invention. It is a flowchart explaining the operation | movement when the storage system as a 1st Embodiment of this invention receives write data. 5 is a flowchart for explaining an operation of executing a conversion process by the storage system as the first embodiment of the invention. It is a block diagram which shows the structure of the storage system as the 2nd Embodiment of this invention. It is a figure which shows an example of the hardware constitutions of the storage system as the 2nd Embodiment of this invention. It is a figure which shows an example of the metadata in the 2nd Embodiment of this invention. It is a figure which shows another example of the metadata in the 2nd Embodiment of this invention. It is a figure which illustrates typically the process in which the selection part selects a data block in the 2nd Embodiment of this invention. It is a flowchart explaining the operation | movement when the access apparatus in the 2nd Embodiment of this invention receives write data. It is a flowchart explaining the operation | movement when the storage node in the 2nd Embodiment of this invention receives a fragment. It is a flowchart explaining the operation | movement which the access apparatus in the 2nd Embodiment of this invention performs a conversion process. It is a block diagram which shows the structure of the storage system as the 3rd Embodiment of this invention. It is a figure which shows an example of the hardware constitutions of the storage system as the 3rd Embodiment of this invention. It is a flowchart explaining the operation | movement which the client apparatus in the 3rd Embodiment of this invention selects a data block. It is a flowchart explaining the operation | movement at the time of the access apparatus in the 3rd Embodiment of this invention receiving a data block.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 shows a functional block configuration of the storage system 1 as the first embodiment of the present invention. In FIG. 1, the storage system 1 includes a selection unit 11, a writing unit 12, a conversion unit 13, and a distributed storage unit 80.

  Here, the storage system 1 can be configured by hardware elements as shown in FIG. In FIG. 2, the storage system 1 includes a CPU (Central Processing Unit) 1001, a memory 1002, a plurality of storage devices 1004, and a network interface 1005. The memory 1002 includes a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The storage device 1004 is configured by an auxiliary storage device such as a hard disk or a flash memory. In this case, the selection unit 11, the writing unit 12, and the conversion unit 13 are configured by a network interface 1005 and a CPU 1001 that reads and executes a computer program stored in the memory 1002 or the storage device 1004. The distributed storage unit 80 includes a plurality of storage devices 1004. However, the hardware configuration of the storage system 1 and each functional block thereof is not limited to the above-described configuration.

  2 shows three storage devices 1004, the number of storage devices 1004 included in this embodiment is not limited. In the storage system 1, capacity and performance can be expanded by adding or replacing a plurality of storage devices 1004 constituting the distributed storage unit 80.

  The distributed storage unit 80 stores data that has been subjected to redundant processing. The redundancy process is a process for dividing data into a plurality of divided data and adding redundant information, for example. The redundant information is information generated based on the error correction code technique so that the original data can be restored even if a part of the divided data is lost. In this case, that is, the data subjected to the redundancy processing includes divided data and redundant information. The distributed storage unit 80 stores the data subjected to the redundancy processing, that is, the divided data and the redundant information, distributed to the plurality of storage devices 1004.

  The selection unit 11 divides the write data into data blocks. The data block to be divided may have a fixed length or a variable length. The write data is data that is received when a write is requested by an external device. For example, the write data may be received from an external device using a file transfer protocol such as NFS (Network File System) or CIFS (Common Internet File System). Note that the write data includes not only newly written data but also data written for updating.

  The selection unit 11 selects a data block to be written to the distributed storage unit 80 from the divided data blocks. Specifically, the selection unit 11 first selects, from among the divided data blocks, a data block in which a data block having the same content is not stored in the distributed storage unit 80 as a write target.

  Further, the selection unit 11 performs distributed storage from among the data blocks in which the data blocks having the same contents among the divided data blocks are already stored in the distributed storage unit 80 and the redundancy does not satisfy the required value. A data block to be written to the unit 80 is selected. At this time, the selection unit 11 may select some, not all, data blocks having the same content already stored in the distributed storage unit 80 and whose redundancy does not satisfy the required value. desirable.

  Here, the redundancy of the data block is information indicating the amount of redundant information in the data block subjected to the redundancy process. For example, the redundancy may be a ratio of redundant information in a data block. Alternatively, the redundancy may be the number of redundant information included in the data block.

  The required value is a value required as the redundancy of write data. For example, as the request value, a value common to each file system realized on the storage system 1 may be set. Such a required value may be stored in the memory 1002 or the storage device 1004. Further, it is assumed that the required value can be changed by setting. Further, that the redundancy satisfies the required value may be that the redundancy matches the required value. Alternatively, that the redundancy satisfies the required value may be that the redundancy is equal to or higher than the required value. In addition, the redundancy satisfying the required value means that the reliability of the data block secured by the redundancy is equal to or higher than the reliability of the data block secured by the redundancy satisfying the requested value. What is necessary is just to become above.

  If it is considered that the required value is satisfied when the redundancy matches the required value, the data block whose redundancy does not satisfy the required value is a data block whose redundancy is smaller than the required value and the required redundancy. Data blocks that are larger than the value and excessive are applicable. These data blocks are converted into data blocks having the redundancy indicated by the request value by the writing unit 12 or the conversion unit 13 described later. In this case, the use capacity of the distributed storage unit 80 is reduced by reducing the data block whose redundancy is larger than the required value to the required redundancy.

  Also, if it is considered that the required value is satisfied when the redundancy is equal to or higher than the required value, the data block whose redundancy does not satisfy the required value corresponds to a data block whose redundancy is smaller than the required value. Data blocks whose value is larger than the required value are not applicable. In this case, the number of data blocks to be processed is further suppressed by the writing unit 12 or the conversion unit 13 described later.

  The writing unit 12 performs redundancy processing based on the redundancy satisfying the requested value on the data block selected by the selection unit 11 and stores the data block in the distributed storage unit 80. Specifically, the writing unit 12 divides the data block into a plurality of fragments. The divided fragments are called original fragments. Then, the writing unit 12 generates one or more pieces of redundant information based on the divided fragments. A fragment generated as redundant information is called a parity fragment. Also, the original fragment and the parity fragment are collectively referred to as a fragment. At this time, it is assumed that the number of original fragments for dividing the data block and the number of parity fragments to be added are determined according to the redundancy. For example, if the redundancy indicates the ratio of the number of parity fragments to the total number of fragments, when the redundancy (the ratio of the number of parity fragments) is changed, the number of original fragments and parity fragments are changed. Become. For example, it is assumed that the redundancy indicates the number of parity fragments, and the total number of fragments is predetermined. In this case, when the redundancy (number of parity fragments) is changed, the numbers of original fragments and parity fragments are changed. Note that various error correction code techniques may be applied to such redundant processing. Then, the writing unit 12 distributes and stores these fragments in the plurality of storage devices 1004 in the distributed storage unit 80.

  Here, FIG. 3 schematically shows data blocks constituting the write data and fragments constituting the data blocks. In FIG. 3, the write data is divided into k data blocks 1 to k. k is a positive integer. Each data block may be fixed length or variable length. The data block 1 is divided into m original fragments OF1 to OFm. m is a positive integer. For the data block 1, n parity fragments PF1 to PFn are generated. n is a positive integer. That is, the data block 1 subjected to the redundancy processing is composed of m original fragments and n parity fragments. When the redundancy is expressed by the number of parity fragments, the redundancy of the data block 1 is n. In addition, when the redundancy is expressed as a parity fragment ratio, it is n / (n + m). “/” Represents division.

  The conversion unit 13 performs the redundancy conversion process for the data block whose redundancy does not satisfy the required value among the data blocks stored in the distributed storage unit 80 while suppressing the processing speed. The redundancy conversion process is a process in which a stored data block is read, converted into a data block subjected to a redundancy process based on a redundancy satisfying a required value, and then stored in the distributed storage unit 80 again. . Specifically, the conversion unit 13 restores the corresponding data block by reading the original fragment and the parity fragment. Then, the conversion unit 13 divides the restored data block into new original fragments based on the redundancy satisfying the required value, and generates a new parity fragment. Then, the conversion unit 13 distributes and stores the newly generated original fragment and parity fragment in the plurality of storage devices 1004 in the distributed storage unit 80.

  The operation of the storage system 1 configured as described above will be described with reference to the drawings.

  First, FIG. 4 shows the operation of the storage system 1 when write data is received from the outside.

  In FIG. 4, first, the selection unit 11 acquires a required value for redundancy (step A1).

  Next, the selection unit 11 divides the write data into data blocks (step A2).

  Next, the selection unit 11 selects, as a write target, a data block in which a data block having the same content is not stored in the distributed storage unit 80 among the divided data blocks (step A3).

  Next, the selection unit 11 selects the data to be written from among the divided data blocks from among the data blocks in which the data blocks having the same contents are stored in the distributed storage unit 80 and the redundancy does not satisfy the required value. A block is selected (step A4).

  Next, the writing unit 12 performs a redundancy process based on the redundancy satisfying the requested value on the data block selected in Step A3 and Step A4 (Step A5).

  Next, the writing unit 12 stores the data block subjected to the redundancy processing in the distributed storage unit 80 (step A6).

  This is the end of the description of the operation when the storage system 1 receives write data from the outside.

  Next, the operation of the redundancy conversion process by the storage system 1 is shown in FIG.

  In FIG. 5, first, the conversion unit 13 extracts a data block whose redundancy does not satisfy the required value from among the data blocks stored in the distributed storage unit 80 as a conversion target (step B1).

  Next, the conversion unit 13 reads the data block to be converted (step B2).

  Next, the conversion unit 13 performs a redundancy process on the read data block based on the redundancy satisfying the requested value (step B3).

  Next, the conversion unit 13 stores the data block subjected to the redundancy process in the distributed storage unit 80 (step B4).

  The conversion unit 13 executes such processing in steps B1 to B4 while suppressing the processing speed.

  This is the end of the description of the operation of the storage system 1.

  Next, effects of the first exemplary embodiment of the present invention will be described.

  The storage system according to the first embodiment of the present invention can efficiently change the redundancy while suppressing the influence on the performance in the storage system having the deduplication function.

  The reason will be described. In the present embodiment, the distributed storage unit stores data that has been subjected to redundancy processing. Then, the selection unit divides the write data into data blocks, and selects a data block to be written from among the divided data blocks. Here, a data block in which a data block having the same content is not stored in the distributed storage unit is selected as a write target. Further, a data block extracted from data blocks in which data blocks having the same contents are already stored in the distributed storage unit and whose redundancy does not satisfy the required value is selected as a write target. This is because the writing unit performs redundancy processing based on the redundancy satisfying the required value on the data block selected by the selection unit and stores the data block in the distributed storage unit. In addition, the conversion unit converts a data block whose redundancy does not satisfy the required value among the data blocks stored in the distributed storage unit into a data block subjected to redundancy processing based on the redundancy satisfying the required value and stores the data block. This is because the conversion process to be performed is executed while suppressing the processing speed.

  With this configuration, in the present embodiment, when the required value of the redundancy is changed, the redundancy processing based on the new redundancy is performed among the data blocks constituting the write data and writing is performed. Only the following data blocks are used. That is, the data blocks to be written are only data blocks extracted from data blocks that are not yet stored in the distributed storage unit and data blocks that have the same contents as the stored data blocks and whose redundancy does not satisfy the required value. In other words, in the present embodiment, when data blocks having the same contents are stored at the time of writing, all of those whose redundancy does not satisfy the new required value are compared with a case where new redundancy processing is performed and stored again. This reduces the number of data blocks to be redone. As a result, the present embodiment does not significantly reduce the write performance even after the required redundancy value is changed.

  Further, in the present embodiment, after the redundancy required value is changed, a new redundancy process is not performed at the time of writing. Based on the redundant processing, the data is stored again. At this time, the present embodiment performs such conversion processing while suppressing the processing speed. As a result, this embodiment has a performance to access data in operation compared to the case where redundancy conversion processing is performed on stored data blocks without changing redundancy at the time of writing. The influence can be prevented. Furthermore, in the present embodiment, the redundancy is changed for some data blocks even at the time of writing. Therefore, even if the processing speed of the conversion process performed separately is suppressed, the redundancy is changed for all the data blocks. Time to complete can be shortened.

  In other words, in this embodiment, when data is written, a part of the data block is written with a new redundancy to improve the efficiency of changing the redundancy, and all the data is converted by a conversion process executed separately. Covers changes in redundancy for blocks. In addition, this embodiment reduces the number of data blocks to be written with a new redundancy when writing information, and does not deteriorate the writing performance, and reduces the speed of the conversion process that is executed separately. The time to completion of the conversion process is shortened without affecting the performance.

(Second Embodiment)
Next, a second embodiment of the present invention will be described in detail with reference to the drawings. Note that, in each drawing referred to in the description of the present embodiment, the same reference numerals are given to the same configuration and steps that operate in the same manner as in the first embodiment of the present invention, and the detailed description in the present embodiment. Description is omitted.

  FIG. 6 shows the configuration of the storage system 2 as the second embodiment of the present invention. In FIG. 6, the storage system 2 includes an access device 200 and a plurality of storage nodes 900. The access device 200 is a device that receives data write and read requests to the storage system 2 and manages data in the storage system 2. The storage node 900 is a node that distributes and stores write data to be read / written in the storage system 2. A plurality of storage nodes 900 constitute a distributed storage unit 90. The storage system 2 can be expanded in capacity and performance by adding or replacing the storage node 900.

  As illustrated in FIG. 6, the access device 200 includes a selection unit 21, a writing unit 22, and a conversion unit 23. The storage node 900 includes a data storage unit 91, a metadata storage unit 92, and a data processing unit 93.

  Here, each device constituting the storage system 2 can be configured by hardware elements as shown in FIG. In FIG. 7, the access device 200 includes a CPU 2001, a memory 2002, a storage device 2004, and a network interface 2005. In this case, the selection unit 21, the writing unit 22, and the conversion unit 23 are configured by a network interface 2005 and a CPU 2001 that reads and executes a computer program stored in the memory 2002 or the storage device 2004. The storage node 900 includes a CPU 9001, a memory 9002, a storage device 9004, and a network interface 9005. In this case, the data storage unit 91 and the metadata storage unit 92 are configured by the storage device 9004. The data processing unit 93 includes a network interface 9005 and a CPU 9001 that reads and executes a computer program stored in the memory 9002 or the storage device 9004. Although three storage nodes 900 are shown in FIG. 7, the number of storage nodes 900 included in the present embodiment is not limited. FIG. 7 shows one access device 200, but the number of access devices 200 included in the present embodiment is not limited. Further, the hardware configuration of each device and each functional block constituting the storage system 2 is not limited to the above-described configuration.

  Next, each functional block included in the storage node 900 will be described.

  The data storage unit 91 stores fragments that constitute a data block that has been subjected to redundancy processing. As described above, the fragment is a generic name for the original fragment and the parity fragment. Data blocks subjected to redundancy processing by the data storage units 91 of the plurality of storage nodes 900 are distributed and stored.

  The metadata storage unit 92 stores metadata regarding data blocks stored by the distributed storage unit 90. The metadata includes information for determining the redundancy of data blocks and the identity of their contents. These pieces of information are referred to by the selection unit 21 or the conversion unit 23.

  Further, the metadata may include a request value for redundancy set for each write data or write data group. In the first embodiment of the present invention, it has been described that the required value of the redundancy is set to a common value for each file system. In the present embodiment, it is assumed that the required value of redundancy can be set individually for each write data or group of write data. If there is no individual required value applied to the corresponding write data, the commonly set required value is applied. In addition, the commonly set request value may be stored in the memory 2002 or the storage device 2004 of the access device 200.

  In addition, the metadata includes various kinds of information necessary for reading and writing the write data. For example, the metadata may include information indicating the logical address of each data block constituting the write data. Further, the metadata may include information indicating the physical addresses of the fragments that constitute the data block. Further, the metadata may include information indicating the number of references of each data block. The reference number is the number of write data that is a reference source of the data block.

  An example of metadata stored in the metadata storage unit 92 is shown in FIGS.

  FIG. 8 is an example of metadata including a logical address of each data block constituting write data. In this example, the write data is represented as a file or directory. Files or directories are managed hierarchically. The metadata about the directory “Root dir fid = 1” and the directory “dirA fid = 3” includes the names and fids of the files or directories under the metadata. fid is information for identifying a file or a directory. The metadata about the file “fileA fid = 2” includes an offset and a logical address of each data block constituting the file. The offset represents the starting position of the data block in the file.

  FIG. 9 is an example of metadata including information for determining the redundancy and identity of data blocks. In this example, metadata is stored in association with the logical address of each data block. Here, the redundancy is represented by the number of parity fragments added to the data block. The hash value is an example of information for determining the identity of the contents of the data block. In this case, the hash value is preferably a value calculated using an irreversible hash function such as MD5 (Message Digest Algorithm 5) and SHA-256 (Secure Hash Algorithm 256-bit). The reference number represents the number of write data (files) referring to the data block as described above. The physical address represents the physical address of each fragment constituting the data block separated by “,”. In this example, the physical address of each fragment includes information for specifying the storage node 900 and information indicating the physical storage position on the data storage unit 91.

  Note that the metadata storage units 92 of the plurality of storage nodes 900 may each store metadata for all data blocks stored in the storage system 2. Alternatively, any one of the metadata storage units 92 of the plurality of storage nodes 900 may collectively store metadata for all data blocks stored in the storage system 2. However, in this case, it is desirable that the metadata storage unit 92 is backed up by another storage node 900. Alternatively, the metadata storage units 92 of the plurality of storage nodes 900 may store metadata about data blocks stored in the storage system 2 in a distributed manner.

  The data processing unit 93 receives information from the access device 200 and writes information stored in the data storage unit 91 or the metadata storage unit 92. Further, the data processing unit 93 reads information stored in the data storage unit 91 or the metadata storage unit 92 based on a request from the access device 200 and transmits the information to the access device 200.

  Next, each functional block included in the access device 200 will be described.

  The selection unit 21 is configured in substantially the same manner as the selection unit 11 in the first embodiment of the present invention, but the details of the process of selecting a data block to be written are different. The selection unit 21 extracts a predetermined amount of data blocks from among data blocks whose data blocks having the same contents have already been stored in the distributed storage unit 90 and whose redundancy does not satisfy the required value, and is selected as a write target To do.

  For example, the selection unit 21 classifies the data blocks constituting the write data into the following three types by verifying the identity of the contents with the stored data blocks and verifying the redundancy. The first type is a data block in which data blocks having the same content are not stored in the distributed storage unit 90. The second type is a data block in which data blocks having the same contents have already been stored in the distributed storage unit 90 and the redundancy satisfies the required value. The third type is a data block in which data blocks having the same contents have already been stored in the distributed storage unit 90 and the redundancy does not satisfy the required value.

  Here, the selection part 21 performs the verification of the identity of the content about each data block as follows. That is, the selection unit 21 calculates information for determining identity for each data block. The selecting unit 21 stores the data block having the same content in the distributed storage unit 90 based on whether or not the information for determining the calculated identity is already stored in the metadata storage unit 92. It is determined whether or not there is.

  Moreover, the selection part 21 performs the verification of the redundancy about each data block as follows. That is, for each data block, if the data block having the same content has already been stored, the selection unit 21 acquires the redundancy from the metadata storage unit 92. Further, the selection unit 21 acquires a required value of redundancy applied to write data including the data block. The redundancy applied to the write data is a request value set individually for the write data or a group to which the write data belongs, or a request value set in common. Then, the selection unit 21 determines whether or not the redundancy of data blocks having the same content satisfies the required value. Thereby, the selection part 21 can classify each data block which comprises write data into the above-mentioned three types. For example, the selection unit 21 may perform the following selection process by temporarily storing the information obtained by classifying the data block into the three types in the memory 1002 or the storage device 1004.

  Then, the selection unit 21 first selects the first type of data block as a write target. Furthermore, the selection unit 21 selects a predetermined amount of data blocks from among the third type data blocks as a write target. The predetermined amount may be the number of data blocks. The predetermined amount may be a ratio to the number of the third type data blocks. The predetermined amount may be a predetermined ratio with respect to the number of data blocks constituting the write data. The predetermined amount may be a total size of data blocks. The predetermined amount may be a predetermined ratio with respect to the total size of the third type data block. Further, the predetermined amount may be a predetermined ratio with respect to the size of the write data. In addition, the predetermined amount may be another amount as long as it represents the amount of data blocks to be selected.

  Further, the selection unit 21 can apply various criteria as criteria for selecting a predetermined amount of data blocks. For example, the selection unit 21 may select up to a predetermined amount of the third type of data blocks in order from the top of the data blocks arranged in the order of constituting the write data.

  A selection process of a predetermined amount of data blocks is schematically shown in FIG. In FIG. 10, a rectangle represents a data block into which write data is divided. Moreover, the rectangle filled with gray represents the selected data block. The k data blocks are arranged from the first to the kth in order from the top of the write data. The numbers in the rectangles represent the above-mentioned classification. That is, a rectangle surrounding 1 represents a first type of data block in which data blocks having the same contents are not stored. The rectangle surrounding 2 represents a second type of data block in which data blocks having the same contents are stored and the redundancy satisfies the required value. The rectangle surrounding 3 represents a third type of data block in which data blocks having the same contents are stored and the redundancy does not satisfy the required value. At this time, all the first type data blocks are selected. Assuming that “M from the top” is defined as the predetermined amount and the selection criterion, the M data blocks from the top are selected for the third type of data block, and the M + 1 and subsequent data blocks are not selected.

  Note that the selection unit 21 may not perform selection after the verification of the identity and redundancy of the contents of all the data blocks constituting the write data. In other words, the selection unit 21 may select a predetermined amount of data blocks based on the above-described selection criteria while verifying the identity and redundancy of the data blocks constituting the write data. For example, the selection unit 21 selects a data block according to the above-described selection criteria every time it finishes verifying the identity and redundancy of a predetermined number of data blocks in order from the top of the data block constituting the write data. May be. Alternatively, the selection unit 21 may select a data block according to the selection criterion described above every time a predetermined period elapses while executing verification of identity and redundancy in order from the top of the data block constituting the write data. . In this case, the selection unit 21 may notify the writing unit 22 described later of the selected data block every time selection processing is performed.

  The writing unit 22 is configured in substantially the same manner as the writing unit 12 in the first embodiment of the present invention. However, the present embodiment is different in that the original fragment and the parity fragment generated by performing the redundancy process on the write data are transmitted to the storage node 900 in order to be distributed and stored. As for which fragment is transmitted to which storage node 900, for example, a method of determining by round robin is applicable. In addition, various known techniques may be applied as to which fragment is transmitted to which storage node 900.

  Note that when the writing unit 22 is notified every time the data block to be written is selected by the selection unit 21, the writing unit 22 performs redundancy processing on the notified data block each time the notification is made, and What is necessary is just to transmit to the node 900.

  The conversion unit 23 is configured in substantially the same manner as the conversion unit 13 in the first embodiment of the present invention, but the details of the technique for suppressing the speed of the conversion process are different. The conversion unit 23 suppresses the processing speed by executing conversion processing for each corresponding data block at each predetermined timing.

  In addition, the conversion unit 23 refers to the metadata storage unit 92 of the storage node 900 in order to extract a data block to be converted from data blocks stored in the distributed storage unit 90. Specifically, the conversion unit 23 compares the redundancy of each data block stored in the metadata storage unit 92 with the required value of the redundancy applied to the write data that refers to the data block. Then, the conversion unit 23 may extract a data block whose redundancy does not satisfy the required value as a target of conversion processing for redundancy.

  The operation of the storage system 2 configured as described above will be described with reference to the drawings.

  First, FIG. 11 shows the operation of the storage system 2 when write data is received from the outside. Here, a hash value is adopted as information for determining identity. Further, it is assumed that “M” is defined as the predetermined amount and the selection criterion “from the top” is defined.

  Here, first, the selection unit 21 acquires a required value of redundancy applied to write data (step A11).

  As described above, the selection unit 21 acquires a request value individually set for the write data or the group or a request value set in common.

  Next, the selection unit 21 divides the write data into data blocks (step A12).

  Next, the selection unit 21 calculates a hash value for each data block (step A13).

  Next, the selection unit 21 verifies the identity of contents and the redundancy for each data block (step A14).

  Specifically, the selection unit 21 determines whether the same hash value as the hash value calculated for each data block is stored in the metadata storage unit 92. Further, the selection unit 21 acquires the redundancy of the data block having the same hash value from the metadata storage unit 92. Then, the selection unit 21 classifies the data blocks that do not store the same hash value as the first type. In addition, the selection unit 21 classifies the data blocks in which the same hash value is stored and the redundancy satisfies the required value as the second type. Further, the selection unit 21 classifies data blocks in which the same hash value is stored and the redundancy does not satisfy the required value as the third type. As described above, that the redundancy satisfies the required value may be that the redundancy matches the required value, or that the redundancy is equal to or higher than the required value.

  Next, the selection unit 21 selects the data block classified into the first type. That is, the selection unit 21 selects all the data blocks that are not stored in the storage node 900 (step A15).

  Next, the selection unit 21 selects a predetermined amount of data blocks from the data blocks classified into the third type (step A16).

  Here, as described above, the selection unit 21 selects up to a predetermined number of third-type data blocks in order from the top of the write data.

  Next, the writing unit 22 performs redundancy processing based on the redundancy satisfying the requested value for the data block selected in steps A15 and A16. Specifically, the writing unit 22 divides each data block based on the redundancy satisfying the required value, generates an original fragment, and generates a parity fragment (step A17).

  Then, the writing unit 22 distributes and transmits information including the original fragment and the parity fragment to the plurality of storage nodes 900 (step A18). At this time, the writing unit 22 transmits metadata including the hash value and redundancy of the data block together with these fragments.

  This is the end of the description of the operation when the access device 200 receives write data from the outside.

  Next, FIG. 12 shows an operation when the storage node 900 receives a fragment.

  In FIG. 12, first, the data processing unit 93 receives a fragment from the access device 200 (step C1).

  Note that one or more fragments are received. In addition, the received fragment may be composed of either one of the original fragment and the parity fragment, or may be composed of both types.

  Next, the data processing unit 93 stores the received fragment in the data storage unit 91 (step C2).

  Next, the data processing unit 93 updates the metadata of the data block including the received fragment (step C3).

  For example, it is assumed that the information shown in FIGS. 8 and 9 is stored in the metadata storage unit 92. If the data block including the received fragment is newly written, the data processing unit 93 provides information including the hash value, redundancy, and physical address regarding the new data block as shown in FIG. Add and store as simple metadata. If the data block including the received fragment has the same contents as before and the redundancy is converted, the data processing unit 93 updates the physical address and the redundancy of the corresponding data block. In addition, if the write data configured by the data block including the received fragment is newly written, the data processing unit 93 sets the information related to the new write data as metadata as illustrated in FIG. Add and remember. If the write data configured by the data block including the received fragment is updated, the data processing unit 93 updates the metadata of the corresponding write data.

  Thus, the storage node 900 ends the operation when receiving the fragment.

  Next, the redundancy conversion processing of the storage system 2 is shown in FIG.

  In FIG. 13, first, the conversion unit 23 acquires the redundancy of one of the data blocks stored in the distributed storage unit 90 from the metadata storage unit 92 (step B11).

  Next, the conversion unit 23 acquires a required value of redundancy applied to write data including the data block (step B12).

  Next, the conversion unit 23 determines whether or not the redundancy of the data block satisfies the required value (step B13).

  Here, when the redundancy satisfies the required value, the operation of the storage system 2 proceeds to Step B17.

  On the other hand, when the redundancy of the data block does not satisfy the required value in step B13, the conversion unit 23 reads the corresponding data block from the distributed storage unit 90 (step B14).

  Specifically, the conversion unit 23 reads the original fragment and the parity fragment from the plurality of storage nodes 900, and restores the corresponding data block based on the read fragment.

  Next, the conversion unit 23 performs redundancy processing based on the redundancy satisfying the requested value on the read data block to generate an original fragment and a parity fragment (step B15).

  Next, the conversion unit 23 distributes the original fragment and the parity fragment to the plurality of storage nodes 900 and transmits them (step B16).

  And if the other data block which has not yet performed the process from step B11 is memorize | stored in the dispersion | distribution memory | storage part 90 (Yes in step B17), the conversion part 23 will wait for predetermined time (step B18), and step Repeat the process from B11.

  On the other hand, if another data block that has not been processed from step B11 is not stored in the distributed storage unit 90 (No in step B17), the storage system 2 ends the redundancy conversion process.

  Next, the effect of the second exemplary embodiment of the present invention will be described.

  The storage system according to the second embodiment of the present invention can change the redundancy more efficiently while suppressing the influence on the performance in the storage system having the deduplication function.

  The reason will be described. In the present embodiment, in the storage node, the data storage unit stores information obtained by dividing the data block. Further, the metadata storage unit stores information for determining the redundancy and identity of the stored data block. Then, when the selection unit of the access device selects a data block to be written from the data blocks constituting the write data, by referring to the metadata storage unit, it is possible to verify the identity of the content and the redundancy Perform verification. Then, the selection unit selects a data block in which a data block having the same content is not yet stored as a write target. In addition, the selection unit selects a predetermined amount of data blocks to be written from data blocks in which data blocks having the same contents are stored and whose redundancy does not satisfy the required value. Then, the writing unit of the access device distributes the data block selected as a writing target and writes it to a plurality of storage nodes. At this time, the storage node updates information for determining the redundancy and identity of the metadata storage unit. In addition, the conversion unit of the access device refers to the redundancy stored in the metadata storage unit, and extracts a data block that is a target of the conversion processing of the redundancy. This is because the conversion unit performs the redundancy conversion process for each data block to be processed at each predetermined timing.

  As described above, according to the present embodiment, when the required value of the redundancy is changed, the redundancy processing based on the redundancy satisfying the new required value is performed among the data blocks constituting the write data and written. Reduce the number of data blocks more appropriately. As a result, the present embodiment does not further effectively reduce the write performance even after the required redundancy value is changed.

  In addition, the present embodiment more appropriately sets the processing speed of the conversion processing to be separately performed on the data block that has not been re-stored without performing new redundancy processing at the time of writing after the required value of redundancy is changed. Suppress. As a result, this embodiment has an effect on the access performance in operation compared to the case where redundancy conversion processing is performed on stored data blocks at once without changing the redundancy at the time of writing. Can be prevented from coming out further. Furthermore, since the redundancy is changed for some data blocks at the time of writing in this embodiment, the redundancy can be changed for all the data blocks even if the speed of conversion processing performed separately is suppressed. Time to complete can be shortened.

  In the present embodiment, the selection unit selects up to a predetermined number of third-type data blocks in order from the top of the write data as a reference for selecting a predetermined amount of data blocks from the third-type data block. The example to do was demonstrated. However, the selection unit may select a predetermined amount of the third type data block in order from the end of the write data. Alternatively, the selection unit may randomly select a predetermined amount of data blocks from the third type of data blocks. Alternatively, the selection unit may select a predetermined amount of data blocks from the third type of data block in descending order of access frequency.

  Alternatively, the selection unit may select a data block by combining a plurality of predetermined amounts and selection criteria. For example, it is assumed that the selection unit first selects a third predetermined data block (for example, up to a predetermined number) according to a certain selection criterion (for example, in order from the top). Further, when the selected data block does not reach the second predetermined amount (for example, a predetermined ratio), the selection unit selects the third type of data block according to another selection criterion (for example, the access frequency). You may make it select in addition (in order of high). However, even in such a case, it is desirable that the selection unit does not select all of the third type data blocks.

  Moreover, in this Embodiment, the conversion part demonstrated the example which suppresses the speed of a conversion process by performing the conversion process with respect to each data block used as the conversion object for every predetermined timing. Not limited to this, the conversion unit sets an upper limit on the number of data blocks to be converted during the unit period, and when the upper limit is exceeded, the conversion unit temporarily stops the conversion process until the next unit period. Thus, the processing speed may be suppressed. Alternatively, the conversion unit may perform the conversion process on a predetermined number of data blocks at predetermined intervals. In addition, the conversion unit may suppress the processing speed by other means.

  In the present embodiment, the example in which the metadata storage unit is provided in the storage node has been described. However, the present invention is not limited to this, and the metadata storage unit may be provided in the access device. The metadata storage unit may be provided in a client device that requests data writing. In this case, the client device may store metadata related to write data written by the own device in the storage node in the metadata storage unit. In this case, the client device may not share the information in the metadata storage unit with other client devices, or may share it.

(Third embodiment)
Next, a third embodiment of the present invention will be described in detail with reference to the drawings. Note that, in each drawing referred to in the description of the present embodiment, the same reference numerals are given to the same configuration and steps that operate in the same manner as in the second embodiment of the present invention, and the detailed description in the present embodiment. Description is omitted.

  FIG. 14 shows the configuration of the storage system 3 as the third embodiment of the present invention. 14, the storage system 3 differs from the storage system 2 according to the second embodiment of the present invention in that it includes an access device 300 instead of the access device 200 and further includes a client device 400. The client device 400 is a device that requests the storage node 900 to write write data via the access device 300.

  As shown in FIG. 14, the access device 300 includes a selection unit 21 and a point provided with a writing unit 32 instead of the writing unit 22 with respect to the access device 200 according to the second embodiment of the present invention. There is no difference. The client device 400 includes a selection unit 41.

  Here, each device constituting the storage system 3 can be constituted by hardware elements as shown in FIG. In FIG. 15, the client device 400 includes a CPU 4001, a memory 4002, a storage device 4004, and a network interface 4005. In this case, the selection unit 41 includes a network interface 4005 and a CPU 4001 that reads and executes a computer program stored in the memory 4002 or the storage device 4004. The access device 300 includes hardware elements similar to those of the access device 200 according to the second embodiment of the present invention. Note that the hardware configuration of each device and each functional block constituting the storage system 3 is not limited to the above-described configuration.

  Next, functional blocks included in the client device 400 will be described first.

  The selection unit 41 is configured in substantially the same manner as the selection unit 21 in the second embodiment of the present invention. However, the selection unit 41 is different in that it operates on write data generated in its own device. The selection unit 41 operates by receiving write data from other functional blocks (not shown) in its own device. The selection unit 41 also differs in that the data block selected as a write target and metadata including information for determining redundancy and identity thereof are transmitted to the access device 300. Note that the selection unit 41 refers to metadata about a stored data block in order to select a data block to be written. At this time, the selection unit 41 may acquire the metadata by requesting the access device 300 to refer to the metadata. Further, the selection unit 41 refers to the required value of redundancy in order to select a data block to be written. At this time, the selection unit 41 may acquire a required value for redundancy from the access device 300.

  Next, functional blocks included in the access device 300 will be described.

  The writing unit 32 is configured in substantially the same manner as the writing unit 22 in the second embodiment of the present invention. However, the writing unit 32 is different in that it operates on a data block received as a writing target from the client device 400.

  In addition, the access device 300 has a function of acquiring metadata stored in the metadata storage unit 92 of the storage node 900 in response to a request from the client device 400 and returning it to the client device 400. Further, the access device 300 has a function of returning a request value for redundancy in response to a request from the client device 400. At this time, the access device 300 may acquire a request value for the redundancy set in common for each data block from the memory 2002 or the storage device 2004 of the own device and return it to the client device 400. Further, the access device 300 may obtain a request value for redundancy individually set for the corresponding data block from the metadata storage unit 92 of the storage node 900 and return it to the client device 400.

  The operation of the storage system 3 configured as described above will be described with reference to the drawings.

  First, FIG. 16 shows an operation when write data is generated in the client device 400.

  In FIG. 16, the selection unit 41 operates in the same manner as the selection unit 21 of the access device 200 in the second embodiment of the present invention from step A11 to A16. Thereby, a data block to be written is selected from the data blocks into which the write data is divided.

  Next, the selection unit 41 transmits information including the selected data block and metadata to the access device 300 (step A27).

  Thus, the client device 400 ends the operation when write data is generated.

  Next, FIG. 17 shows an operation when the access device 300 receives a data block from the client device 400.

  In FIG. 17, the writing unit 32 first receives information including the selected data block from the client device 400 (step A28).

  Thereafter, the writing unit 32 operates in the same manner as in the second embodiment of the present invention from step A17 to A18. Thereby, each data block selected as a write target is transmitted to a plurality of storage nodes 900 as an original fragment and a parity fragment that have been subjected to redundancy processing based on the redundancy satisfying the requested value.

  As described above, the access device 300 ends the operation when the data block is received from the client device 400.

  The operation when the storage node 900 receives a fragment is the same as the operation of the second embodiment of the present invention described with reference to FIG.

  Further, the operation of the conversion process performed by the access device 300 at every predetermined timing is the same as the operation of the second embodiment of the present invention described with reference to FIG. Omitted.

  Next, effects of the third exemplary embodiment of the present invention will be described.

  The storage system according to the third embodiment of the present invention has a deduplication function, and reduces the amount of communication from the client in the storage system that efficiently changes the redundancy while suppressing the influence on the performance. be able to.

  The reason will be described. In the present embodiment, the selection unit in the client device divides the write data generated in its own device into data blocks, and the data block to be written is selected from the data blocks as in the second embodiment of the present invention. Select similarly. Then, the selection unit transmits the selected data block to the access device. This is because the writing unit in the access device performs redundancy processing on the received data block and transmits it to the storage node.

  As a result, according to the present embodiment, it is possible to reduce the amount of communication from the client device that requests writing of write data to the access device, and to reduce the load on the network and the access device.

  In each of the embodiments of the present invention described above, the redundancy has been described mainly with an example expressed by the number of parity fragments or the ratio of the number. For example, the redundancy may be expressed by a combination of the number of parity fragments and the number of original fragments. In addition, the redundancy may be represented by other information as long as the information represents the amount of redundant information in the data block.

  Further, in each of the embodiments of the present invention described above, the description has focused on an example in which each functional block of the storage system is realized by a CPU that executes a computer program stored in a memory or a storage device. However, the present invention is not limited to this, and some, all, or a combination of each functional block may be realized by dedicated hardware.

  Further, in each of the above-described embodiments of the present invention, each functional block of the storage system may be further distributed to a plurality of devices.

  In each embodiment of the present invention described above, the operation of the storage system described with reference to each flowchart is stored in the data storage unit (storage medium) of the computer device as the computer program of the present invention. Then, the computer program may be read and executed by the CPU. In such a case, the present invention is constituted by the code of the computer program or a storage medium.

  Moreover, each embodiment mentioned above can be implemented in combination as appropriate.

  The present invention is not limited to the above-described embodiments, and can be implemented in various modes.

A part or all of each of the above-described embodiments can be described as in the following supplementary notes, but is not limited thereto.
(Appendix 1)
A distributed storage unit that stores the data subjected to the redundant processing in a distributed manner in a plurality of storage devices;
Write data requested to be written is divided into data blocks, and among the divided data blocks, a data block having the same content data stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit A data block extracted from the data blocks that have been stored in the memory and whose redundancy does not satisfy the required value;
For the data block selected by the selection unit, a writing unit that performs redundancy processing based on redundancy satisfying the required value and stores the data in the distributed storage unit;
Of the data blocks stored in the distributed storage unit, the data block whose redundancy does not satisfy the required value is converted into a data block subjected to redundancy processing based on the redundancy satisfying the required value, A conversion unit that executes conversion processing to be stored in the distributed storage unit while suppressing processing speed;
Storage system with
(Appendix 2)
The selection unit extracts a predetermined amount of data blocks from among data blocks whose data blocks having the same contents are already stored in the distributed storage unit and whose redundancy does not satisfy the required value, and is used as the write target. The storage system according to appendix 1, wherein the storage system is selected.
(Appendix 3)
The storage system according to appendix 1 or appendix 2, wherein the conversion unit suppresses the processing speed by executing the conversion process at every predetermined timing.
(Appendix 4)
For each data block stored in the distributed storage unit, further comprising a metadata storage unit storing metadata including information for determining the redundancy and the identity of the content,
The selection unit refers to the metadata storage unit, and for the data block, whether or not a data block having the same content has already been stored in the distributed storage unit, and its redundancy indicates the required value. 4. The storage system according to any one of appendix 1 to appendix 3, wherein whether or not it is satisfied is determined.
(Appendix 5)
An access device including the selection unit, the writing unit, and the conversion unit;
The distributed storage unit;
The storage system according to any one of appendix 1 to appendix 4, further comprising:
(Appendix 6)
A client device having the selection unit;
An access device having the writing unit and the converting unit;
The distributed storage unit;
The storage system according to any one of appendix 1 to appendix 4, further comprising:
(Appendix 7)
The access device according to appendix 5 or appendix 6.
(Appendix 8)
The client device according to attachment 6.
(Appendix 9)
Computer equipment
Using a distributed storage unit that distributes and stores data subjected to redundant processing in a plurality of storage devices,
Write data requested to be written is divided into data blocks, and among the divided data blocks, a data block having the same content data stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit Selected from the data blocks that have been stored in the data block and whose redundancy does not satisfy the required value,
The selected data block is subjected to redundancy processing based on redundancy satisfying the required value, and is stored in the distributed storage unit,
Of the data blocks stored in the distributed storage unit, the data block whose redundancy does not satisfy the required value is converted into a data block subjected to redundancy processing based on the redundancy satisfying the required value, A method of executing conversion processing stored in a distributed storage unit while suppressing processing speed.
(Appendix 10)
Using a distributed storage unit that distributes and stores data subjected to redundant processing in a plurality of storage devices,
Write data requested to be written is divided into data blocks, and among the divided data blocks, a data block having the same content data stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit A program that causes a computer device to execute a selection step of selecting a data block extracted from data blocks that have already been stored in the memory and whose redundancy does not satisfy the required value as a write target.
(Appendix 11)
Using a distributed storage unit that distributes and stores data subjected to redundant processing in a plurality of storage devices,
Write data requested to be written is divided into data blocks, and among the divided data blocks, a data block having the same content data stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit Selecting a data block extracted from data blocks that have been stored in the memory and whose redundancy does not satisfy the required value,
For the data block selected in the selection step, a writing step of performing redundancy processing based on redundancy satisfying the required value and storing the data in the distributed storage unit;
Of the data blocks stored in the distributed storage unit, the data block whose redundancy does not satisfy the required value is converted into a data block subjected to redundancy processing based on the redundancy satisfying the required value, A conversion step for executing the conversion process to be stored in the distributed storage unit while suppressing the processing speed;
That causes a computer device to execute the program.

1, 2, 3 Storage system 11, 21, 41 Selection unit 12, 22, 32 Writing unit 13, 23 Conversion unit 80, 90 Distributed storage unit 91 Data storage unit 92 Metadata storage unit 93 Data processing unit 200, 300 Access device 400 Client device 900 Storage node 1001, 2001, 4001, 9001 CPU
1002, 2002, 4002, 9002 Memory 1004, 2004, 4004, 9004 Storage device 1005, 2005, 4005, 9005 Network interface

Claims (8)

A distributed storage unit that stores the data subjected to the redundant processing in a distributed manner in a plurality of storage devices;
Write data requested to be written is divided into data blocks, and among the divided data blocks, a data block having the same content data stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit A data block extracted from the data blocks that have been stored in the memory and whose redundancy does not satisfy the required value;
For the data block selected by the selection unit, a writing unit that performs redundancy processing based on redundancy satisfying the required value and stores the data in the distributed storage unit;
Of the data blocks stored in the distributed storage unit, the data block whose redundancy does not satisfy the required value is converted into a data block subjected to redundancy processing based on the redundancy satisfying the required value, A conversion unit that performs conversion processing to be stored in the distributed storage unit while suppressing the processing speed by providing an upper limit value for a data block processed per unit time ; and
Storage system with   The selection unit extracts a predetermined amount of data blocks from among data blocks whose data blocks having the same contents are already stored in the distributed storage unit and whose redundancy does not satisfy the required value, and is used as the write target. The storage system according to claim 1, wherein the storage system is selected.   The storage system according to claim 1 or 2, wherein the conversion unit suppresses the processing speed by executing the conversion processing at predetermined timings. For each data block stored in the distributed storage unit, further comprising a metadata storage unit storing metadata including information for determining the redundancy and the identity of the content,
The selection unit selects the data block to be written by referring to the metadata storage unit,
The said conversion part extracts the data block used as the object of the said conversion process from the said distributed storage part by referring to the said metadata storage part, The any one of Claims 1-3 characterized by the above-mentioned. The storage system described in. An access device including the selection unit, the writing unit, and the conversion unit;
The distributed storage unit;
The storage system according to any one of claims 1 to 4, further comprising: A client device having the selection unit;
An access device having the writing unit and the converting unit;
The distributed storage unit;
The storage system according to any one of claims 1 to 4, further comprising:   The access device according to claim 5 or 6. Computer equipment
Using a distributed storage unit that distributes and stores data subjected to redundant processing in a plurality of storage devices,
Write data requested to be written is divided into data blocks, and among the divided data blocks, a data block having the same content data stored in the distributed storage unit and a data block having the same content stored in the distributed storage unit Selected from the data blocks that have been stored in the data block and whose redundancy does not satisfy the required value,
The selected data block is subjected to redundancy processing based on redundancy satisfying the required value, and is stored in the distributed storage unit,
Of the data blocks stored in the distributed storage unit, the data block whose redundancy does not satisfy the required value is converted into a data block subjected to redundancy processing based on the redundancy satisfying the required value, A method of executing conversion processing to be stored in a distributed storage unit while suppressing the processing speed by providing an upper limit value for a data block processed per unit time .

Images