JP2022175713A - Storage device, information processing system, information processing method and program - Google Patents

Abstract

To provide a technology capable of improving the write performance.SOLUTION: When feature data corresponding to first data that is a management unit contained in first write data is to be stored, feature data corresponding to second data that is a management unit subsequent to the first data is stored in association with the feature data corresponding to the first data. When overlap determination is to be performed on second write data, the overlap determination is performed using the feature data corresponding to the first data, the feature data corresponding to the second data is read on the basis of the association with the feature data corresponding to the first data, and the overlap determination is performed using the feature data corresponding to the second data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a storage device, an information processing system, an information processing method, and a program.

Patent Literature 1 discloses a computer system that performs deduplication processing. The host computer of this computer system performs duplication determination processing on all chunks that constitute update content, and updates the content management table. Then, the host computer creates a deduplication list based on the latest content data during the next deduplication process. As a result, the number of chunks transmitted from the host device to the storage device can be reduced.

WO2015/040711

By the way, in a storage device that performs deduplication, when checking whether write data overlaps with data already stored in the storage device, write data is divided into data in a predetermined management unit (for example, chunks of a certain length). ), and feature data such as a hash value calculated from the data in the management unit is compared with feature data already stored in the storage device. Such feature data must be stored for each data in a predetermined management unit in all write data, resulting in an enormous amount of data. As a result, the load of specifying feature data to be used for duplication determination from among the huge amount of feature data stored in the storage device is heavy, and it may be difficult to improve the write performance.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a storage device, an information processing system, an information processing method, and a program that solve the above problems.

In order to achieve the above object, according to a first aspect of the present invention, a storage device associates feature data corresponding to each piece of data in a predetermined management unit included in a bundle or a series of write data. and when the feature data corresponding to the first data of the management unit included in the first write data are stored in the storage device, the feature data included in the first write data and subsequent to the first data. a feature data registering means for storing in the storage means the feature data corresponding to the second data of the management unit in which the feature data corresponds to the first data in association with the feature data corresponding to the first data; and duplication determination for the second write data is performed, duplicate determination is performed using the feature data corresponding to the first data, and the feature data corresponding to the second data is read based on association with the feature data corresponding to the first data. and determination processing means for performing overlap determination using the feature data corresponding to the second data.

To achieve the above object, according to a second aspect of the present invention, an information processing system includes a host and a storage device communicably connected to the host. The storage device comprises: write command processing means for associating characteristic data corresponding to each piece of data in a predetermined management unit included in a set or a series of write data; When the feature data corresponding to the first data is stored in the storage means, the feature data corresponding to the second data of the management unit, which is included in the first write data and succeeds the first data, is stored in the storage means. a feature data registering means for storing in the storage means in association with the feature data corresponding to the first data; and using the feature data corresponding to the first data when determining duplication for the second write data. read out the feature data corresponding to the second data based on the association with the feature data corresponding to the first data; and perform overlap determination using the feature data corresponding to the second data. and determination processing means for performing.

In order to achieve the above object, according to a third aspect of the present invention, an information processing method provides, for each data of a predetermined management unit included in a set or a series of write data, feature data corresponding to the data. When storing the feature data corresponding to the first data of the management unit included in the first write data, the first data of the management unit included in the first write data and subsequent to the first data is stored. When the feature data corresponding to the second data is stored in association with the feature data corresponding to the first data, and the duplication determination is performed for the second write data, the feature data corresponding to the first data is stored. to determine duplication, read out the feature data corresponding to the second data based on the association with the feature data corresponding to the first data, and use the feature data corresponding to the second data to determine duplication including making decisions.

In order to achieve the above object, according to the fourth aspect of the present invention, a program is provided in a computer for each data in a predetermined management unit included in a set or a series of write data, characteristic data corresponding to the data and when storing the feature data corresponding to the first data of the management unit included in the first write data, the management unit that is included in the first write data and succeeds the first data and storing the feature data corresponding to the second data in association with the feature data corresponding to the first data; duplicate determination is performed using the feature data corresponding to the first data, the feature data corresponding to the second data is read out based on the association with the feature data corresponding to the first data, and the feature data corresponding to the second data is read. and performing a duplicate determination using .

According to the present invention, it is possible to reduce the load of specifying feature data used for duplication determination, and to improve write performance.

1 is a schematic configuration diagram showing an information processing system according to an embodiment of the present invention; FIG. It is a figure which shows an example of the content of the chunk address translation table by one Embodiment of this invention. FIG. 4 is a diagram showing an example of the contents of a hash table according to one embodiment of the present invention; FIG. 4 illustrates an example of the contents of a hash entry according to one embodiment of the present invention; It is a figure which shows an example of the content of the hash value determination result information by one Embodiment of this invention. FIG. 4 is a diagram showing an example of content of subsequent hash information according to an embodiment of the present invention; 4 is a flow chart showing the flow of write processing according to an embodiment of the present invention; 4 is a flowchart showing details of the flow of hash value determination processing according to an embodiment of the present invention; 4 is a flow chart showing the flow of read processing according to an embodiment of the present invention; 1 is a diagram showing a storage device with a minimum configuration according to one embodiment of the present invention; FIG. 4 is a flow chart showing the flow of processing in the storage device with the minimum configuration according to one embodiment of the present invention; 1 is a schematic configuration diagram showing the configuration of a computer according to one embodiment of the present invention; FIG.

An information processing system including a storage device according to an embodiment of the present invention will be described below with reference to the drawings.

<1. Configuration of information processing system>
FIG. 1 is a schematic configuration diagram showing an information processing system according to this embodiment. The information processing system S includes, for example, a disk array device 1 and a host 2 . The disk array device 1 is an example of a "storage device".

The disk array device 1 is a storage device capable of storing data in a non-volatile manner. The disk array device 1 performs processing such as writing, reading, or erasing data based on control commands from the host 2 . For example, when receiving a write command from the host 2 , the disk array device 1 writes write data corresponding to the write command to the storage unit M inside the disk array device 1 . When the disk array device 1 receives a read command from the host 2 , it reads the data stored in the storage unit M inside the disk array device 1 and transmits it to the host 2 . The disk array device 1 can be used, for example, in a system in which the same data is copied multiple times, such as in a backup environment or virtual machine environment. The disk array device 1 will be described later in detail.

The host 2 is communicably connected to the disk array device 1 via, for example, a communication cable. The host 2 is composed of one or more host devices 21, for example. Each host device 21 may be a server device, a personal computer, or another computer. Write commands and read commands in the following description mean write commands and read commands sent from the host 2 to the disk array device 1 unless otherwise specified.

<2. Disk array device>
Next, the disk array device 1 will be explained.
As shown in FIG. 1, the disk array device 1 includes, for example, an I/O processing unit (input/output processing unit) 11, a memory 12, a deduplication volume 13, a pool volume 14, and a management volume 15. . In this embodiment, the deduplication volume 13, the pool volume 14, and the management volume 15 constitute the storage unit M. FIG. The storage unit M is an example of a "storage means".

The I/O processing unit 11 manages writing, reading, and erasing of data with respect to the deduplication volume 13 and the pool volume 14 . For example, the I/O processing unit 11 receives write commands and read commands from the host 2 . The I/O processing unit 11 returns a response to the host 2 when the write processing corresponding to the write command or the read processing corresponding to the read command is completed. In this embodiment, the I/O processing unit 11 includes a write command processing unit 111 , a read command processing unit 112 , a hash value determination processing unit 113 and a duplicate elimination processing unit 114 .

When the I/O processing unit 11 receives a write command to the deduplication volume 13, the write command processing unit 111 divides the write data for which write processing is requested by the write command into chunks of a certain length. A chunk is an example of "predetermined management unit data". Write data having a size larger than the size of the chunk is an example of "a group of write data".

In this embodiment, the write command processing unit 111 has a hash value calculation unit 1111 . The hash value calculator 1111 calculates a hash value from each chunk divided by the write command processor 111 and stores the calculated hash value in the memory 12 . A hash value is an example of "feature data." Calculating a hash value from each chunk is an example of associating feature data with each chunk. However, the feature data is not limited to hash values, and may be specific information registered according to each chunk. The write command processing unit 111 is an example of "write command processing means".

When the I/O processing unit 11 receives a read command, the read command processing unit 112 reads the data of each chunk to be read by the read command based on the chunk address conversion table 151 (see FIG. 2), and sends the data to the host 2. Return data. The chunk address conversion table 151 is a table in which information indicating the storage position of data of each chunk is registered. Hereinafter, the data of each chunk may be referred to as "chunk data", the storage location of chunk data may be referred to as "chunk storage location", and information indicating the chunk storage location may be referred to as "chunk storage location information". Sometimes. For convenience of explanation, "chunk storage location" may be used in context to mean "chunk storage location information".

FIG. 2 is a diagram showing an example of the contents of the chunk address conversion table 151. As shown in FIG. The chunk address conversion table 151 includes, for example, a chunk address (deduplicated LD chunk address) stored in the deduplicated volume 13, which will be described later, and a storage position (chunk storage position) is a table that stores a list of correspondence relationships. "LD" means a logical disk. The chunk storage location includes, for example, a chunk storage LDN indicating a logical disk number (LDN) where the chunk is stored, and a chunk storage address indicating the address where the chunk is stored. For example, the chunk storage location includes the volume number of pool volume 14 and the address on pool volume 14 . The chunk address conversion table 151 is generated by the deduplication processing unit 114, which will be described later.

When the write command is received and the write command processing unit 111 divides the write data into chunks and calculates the hash value, the hash value determination processing unit 113 determines all chunks in order from the top chunk of the write range (write data). Whether or not the hash value calculated by the hash value calculation unit 1111 is registered in the hash table 152 (see FIG. 3) is determined, and if the hash value is registered, the hash value is registered. The location information of the hash entry 1521 (see FIG. 3) is stored in the memory 12 . In this embodiment, the hash value determination processing unit 113 stores in the memory 12 the hash value of the succeeding chunk and the information indicating the storage position of the hash value of the succeeding chunk for the chunk for which the hash value is registered. The hash value determination processing unit 113 is an example of "determination processing means". This content will be described in detail below.

The hash value determination processing unit 113 includes, for example, a hash value determination method determination unit 1131, a hash value search unit 1132, and a subsequent hash value comparison unit 1133.

The hash value determination method determination unit 1131 refers to the memory 12 to determine whether or not to pre-acquire subsequent hash values, which will be described later. A “subsequent hash value” is a hash value corresponding to a chunk subsequent to a certain chunk (for example, a chunk of the next address). In the present embodiment, the subsequent hash value is performed with respect to a chunk (for example, a chunk one ahead) ahead of the chunk to be processed by the hash value determination processing unit 113 (hereinafter sometimes referred to as “processing chunk”). obtained in advance during the process.

In this embodiment, the hash value determination method determining unit 1131 determines that the subsequent hash value has not been pre-acquired when the subsequent hash value is not stored in the memory 12 . In this case, the hash value determination method determination unit 1131 selects hash value determination by the hash value search unit 1132 as the hash value determination method. On the other hand, if the subsequent hash value is stored in the memory 12, the hash value determination method determination unit 1131 determines that the subsequent hash value has been pre-acquired. In this case, the hash value determination method determination unit 1131 selects hash value determination by the subsequent hash value comparison unit 1133 as the hash value determination method.

The hash value search unit 1132 reads a hash table 152 (see FIG. 3), which will be described later, to check whether the hash value of the chunk to be processed by the hash value determination processing unit 113 (the chunk being processed) is registered. Investigate and determine whether or not the hash value is registered.

FIG. 3 is a diagram showing an example of the contents of the hash table 152. As shown in FIG. The hash table 152 is a table that stores all hash values corresponding to chunk data already stored in the disk array device 1 . The hash table 152 needs to store hash values corresponding to all chunks of write data, and becomes enormous. On the other hand, the hash table 152 employs an algorithm using hierarchical management so that a desired hash value can be searched at high speed while keeping the table size as small as possible. The hash table 152 is an example of "table information".

As shown in FIG. 3, the hash table 152 includes, for example, information for searching for hash values (eg, tree search information TS) and hash entries 1521 that store information related to hash values. In the example shown in FIG. 3, the hash table 152 includes three layers of tree search information TS (TS1 to TS3).

The tree search information TS includes position information of the tree search information TS to be read next when searching for a hash value. You can check whether the hash value is registered or not. The position information of the hash entry 1521 in which the hash value is registered is stored in the lowest tree search information TS3. Therefore, if the hash value is registered, the location information of the hash entry 1521 can also be acquired.

FIG. 4 is a diagram showing an example of the contents of the hash entry 1521. As shown in FIG. A hash value, a subsequent hash value, a subsequent hash entry position, and a chunk storage position are registered in the hash entry 1521 . The subsequent hash entry position is position information of the hash entry 1521 corresponding to the subsequent hash value. The chunk storage location includes, for example, a chunk storage LDN indicating the logical disk number where the chunk is stored, and a chunk storage address indicating the address where the chunk is stored.

When the hash value of the chunk under processing is registered in the hash table 152, the hash value search unit 1132 determines whether the hash value is registered and the position information of the hash entry 1521 in which the hash value is registered. It is stored in the memory 12 as part of the result information 121 (see FIG. 5). Furthermore, after acquiring the position information of the hash entry 1521 of the chunk in process, the hash value search unit 1132 acquires the subsequent hash value and the subsequent hash entry position by reading the hash entry 1521 (that is, acquires in advance). . The hash value search unit 1132 saves information indicating subsequent hash values and subsequent hash entry positions acquired in the process performed on the chunk in process in the memory 12 as subsequent hash information 122 (see FIG. 6).

FIG. 5 is a diagram showing an example of the content of hash value determination result information 121 corresponding to one piece of write data. The hash value determination result information 121 is generated for each write data and stored in the memory 12 . In the hash value determination result information 121, the chunk number within the write range, the hash value, whether or not the hash value is registered in the hash table 152, and the position information of the hash entry 1521 are associated with each write data. Registered.

FIG. 6 is a diagram showing an example of the content of the subsequent hash information 122 corresponding to one in-process chunk. Subsequent hash information 122 is generated for each in-process chunk and stored in memory 12 . In the subsequent hash information 122, the subsequent hash value and the subsequent hash entry position acquired in advance during the process executed for the chunk in process are registered in association with each other.

The subsequent hash value comparison unit 1133 compares the hash value of the chunk to be processed by the hash value determination processing unit 113 (the chunk being processed) and the hash value obtained in advance in the processing performed for the chunk one ahead of the chunk being processed. is compared with the subsequent hash value stored in the memory 12 (the hash value included in the subsequent hash information 122). If the hash value of the chunk under processing and the subsequent hash value match, the subsequent hash value comparison unit 1133 reads the hash entry 1521 of the hash value (the position of the subsequent hash entry included in the subsequent hash information 122), and It is determined whether the hash entry 1521 of the hash value is valid. The subsequent hash value comparison unit 1133 determines that the hash value is registered if the hash entry 1521 is valid. On the other hand, if the hash entry 1521 is not valid, or if the two hash values compared above do not match, the subsequent hash value comparison unit 1133 performs hash value determination by the hash value search unit 1132 .

If hash entry 1521 is valid, subsequent hash value comparison section 1133 stores predetermined information in memory 12 in the same manner as hash value search section 1132 . That is, the subsequent hash value comparison unit 1133 stores the presence or absence of hash value registration and the position information of the hash entry 1521 in which the hash value is registered as part of the hash value determination result information 121 (see FIG. 5) in the memory 12. Save to Furthermore, after obtaining the position information of the hash entry 1521 of the chunk in process, the subsequent hash value comparison unit 1133 reads the hash entry 1521 to obtain the subsequent hash value and the subsequent hash entry position. The subsequent hash value comparison unit 1133 saves the information indicating the subsequent hash value and the subsequent hash entry position acquired in the process performed for the chunk in process in the memory 12 as the subsequent hash information 122 (see FIG. 6). .

The deduplication processing unit 114 performs deduplication write processing based on the hash value determination result information 121 already stored in the memory 12 in order from the head chunk of the write range (write data). For example, the deduplication processing unit 114 determines that a chunk whose hash value has already been registered in the hash value determination result information 121 is duplicated data, and acquires chunk storage location information from the hash entry 1521 in which the hash value is registered. do. Then, based on the chunk storage location information acquired from the hash entry 1521, the deduplication processing unit 114 registers the chunk storage location information in the chunk address conversion table 151 (see FIG. 2). On the other hand, the deduplication processing unit 114 determines that a chunk whose hash value is not registered in the hash value determination result information 121 is non-duplicated data, and writes the data of the chunk. Then, the deduplication processing unit 114 registers chunk storage location information in the chunk address conversion table 151 according to the chunk storage location where the chunk data is stored.

Specifically, the deduplication processing unit 114 includes a hash value registration unit 1141 and a chunk data write unit 1142 .

The hash value registration unit 1141 stores the hash value, the subsequent hash value, the position of the subsequent hash entry, and the chunk in the new hash entry 1521 of the hash table 152 for the non-duplicate chunk whose hash value is not registered in the hash value determination result information 121. Register storage location information. At this time, the hash value registration unit 1141 acquires the subsequent hash value and the subsequent hash entry position from the subsequent hash information 122 stored in the memory 12 by the hash value determination processing unit 113 . For example, in the example shown in FIG. 8, the hash value registration unit 1141 registers BBBBBB, which is the hash value of chunk number 0001, as the subsequent hash value of the hash entry 1521 in which the hash value AAAAA of chunk number 0000 is registered. Hash value registration unit 1141 is an example of a “characteristic data registration unit”. Registering the subsequent hash value together with the hash entry 1521 in which the hash value is registered means that the feature data corresponding to the second data is stored in the storage unit M in association with the feature data corresponding to the first data. is an example.

The chunk data write unit 1142 writes data of chunks determined to be non-overlapping to the pool volume 14 . The write destination here is the chunk storage position registered in the chunk address conversion table 151 and the hash entry 1521 described above.

The memory 12 is a volatile storage device that temporarily stores various types of control information (for example, hash value determination result information 121 and subsequent hash information 122). The control information (for example, the hash value determination result information 121 and the subsequent hash information 122) stored in the memory 12 in the write process becomes unnecessary when the write process is completed. In this case, the control information saved in the write process is erased from the memory 12 . As a result, the area in the memory 12 in which the control information saved in the write process has been saved becomes an area that can be used for other purposes.

The deduplication volume 13 is a deduplication storage area (for example, a logical disk) accessible by the host 2 . The deduplication volume 13 stores the address of each chunk in each write data (deduplication LD chunk address). The deduplication volume 13 includes, for example, multiple deduplication volumes 131 to which different volume numbers are assigned.

The pool volume 14 is a storage area (for example, a logical disk) that stores the data saved in the deduplication volume 13 in chunks of a certain length. The pool volume 14 is readable and writable by the I/O processor 11 . The pool volume 14 includes, for example, multiple pool volumes 141 to which different volume numbers are assigned.

The management volume 15 is a storage area (eg, logical disk) in which the chunk address conversion table 151 and hash table 152 are stored. The management volume 15 is readable and writable by the I/O processor 11 . The management volume 15 includes, for example, multiple management volumes 151 to which different volume numbers are assigned.

<3. Process Flow>
<3.1 Flow of Write Processing>
Write processing of the deduplication volume 13 will be described with reference to FIGS. 7 and 8. FIG.
FIG. 7 is a flowchart showing the flow of write processing for the deduplication volume 13. As shown in FIG. First, the I/O processor 11 of the disk array device 1 receives a write command and write data corresponding to the write command from the host 2 (S500). When the I/O processing unit 11 receives a write command from the host 2, the write command processing unit 111 divides write data for which write processing is requested by the write command into chunk units (S501). Then, the hash value calculation unit 1111 calculates a hash value for each divided chunk and stores the calculated hash value in the memory 12 (S502).

Subsequently, the hash value determination processing unit 113 determines whether or not each hash value stored in the memory 12 has been stored in the hash table 152 (that is, whether or not there is a registration in the hash table 152). A value determination process is performed (S503). Details of the hash value determination process will be described with reference to FIG.

FIG. 8 is a flowchart showing the details of the flow of hash value determination processing. In the hash value determination process, hash values corresponding to chunks in the write range (hash values stored in the memory 12) are determined one by one in order from the first chunk. First, the hash value determination method determination unit 1131 refers to the memory 12, and acquires the subsequent hash value in the hash value determination process performed in the chunk one ahead of the chunk to be processed for hash value determination (chunk in process). It is checked whether or not it is performed (S600). That is, the hash value determination method determination unit 1131 determines whether or not the subsequent hash information 122 for the in-process chunk is stored in advance in the memory 12 .

The hash value determination method determination unit 1131 selects hash value determination using the subsequent hash value when the subsequent hash value corresponding to the chunk being processed has been obtained (that is, when the subsequent hash information 122 exists, S600: YES). . On the other hand, if the subsequent hash value corresponding to the chunk being processed has not been acquired (that is, if the subsequent hash information 122 does not exist, S600: NO), the hash value determination method determination unit 1131 performs hash determination by hash table search. select. For the first chunk in the write range, since subsequent hash values have not been acquired, hash value determination by hash table search is always selected.

When the hash value determination method determination unit 1131 selects hash value determination by hash table search (S600: NO), the hash value search unit 1132 searches the hash table 152 of the management volume 15 to check whether or not the hash value is registered. investigate. In this embodiment, the hash value search unit 1132 reads the tree search information TS in order from the highest hierarchy of the hash table 152 to search for hash values. The tree search information TS stores information about the read location of the next hierarchy corresponding to the hash value to be searched. Therefore, by reading up to the lowest layer (up to the third layer TS3 in the example shown in FIG. 3), it is determined whether or not the hash value is registered, and the position information of the hash entry 1521 in which the hash value is registered is acquired. (S601-S603).

The hash value search unit 1132 determines whether or not the hash value is registered in the hash table 152 by searching for the hash value described above (S604). If the hash value is registered (S604: YES), the hash value search unit 1132 reads the hash entry 1521 in which the hash value is registered (S605), and acquires the subsequent hash value and the subsequent hash entry position (S606). ). In this case, the hash value search unit 1132 stores the acquired subsequent hash value and subsequent hash entry position in the memory 12 as the next subsequent hash information 122 .

On the other hand, if the hash value determination method determination unit 1131 selects hash value determination based on the subsequent hash value (S600: YES), the subsequent hash value comparison unit 1133 determines whether the subsequent hash value included in the subsequent hash information 122 stored in the memory 12. Then, the subsequent hash value received is compared with the hash value of the chunk being processed (S611). Then, the subsequent hash value comparison unit 1133 determines whether or not the two hash values thus compared match (S612).

If these hash values match (S612: YES), the subsequent hash value comparison unit 1133 removes the hash entry 1521 existing at the subsequent hash entry position included in the subsequent hash information 122 stored in the memory 12 from the management volume 15. Read (S613). Then, the subsequent hash value comparison unit 1133 determines whether or not the read hash entry 1521 is valid (S614). For example, the succeeding hash value comparison unit 1133 can determine the validity of the hash entry 1521 based on whether the hash value registered in the hash entry 1521 matches the hash value of the chunk being processed.

If the hash entry 1521 is valid (S614: YES), the subsequent hash value comparison unit 1133 determines that the hash value is registered, and compares the subsequent hash value registered in the hash entry 1521 and the subsequent hash entry position. It is acquired in the memory 12 (S606). That is, the subsequent hash value comparison unit 1133 stores the acquired subsequent hash value and subsequent hash entry position in the memory 12 as the next subsequent hash information 122 .

On the other hand, if the hash entry 1521 is invalid (S614: NO), or if the subsequent hash value included in the subsequent hash information 122 and the hash value of the chunk being processed do not match (S612: NO), hash value search is performed. The unit 1132 searches the above hash table 152 (S601 to S606) to determine whether or not the hash value is registered.

When it is determined whether or not the hash value of the chunk being processed is registered, the hash value determination processing unit 113 determines whether or not the hash value of the chunk being processed is registered, and if there is registration, the information indicating the hash entry position as the hash value. It is stored in the memory 12 as part of the determination result information 121 (S607). The hash value determination processing unit 113 completes the hash value determination processing when the memory 12 stores the presence/absence of hash value registration for all chunks in the write range (S608: YES).

Next, referring back to FIG. 5, the rest of the write process will be described.
When the hash value determination process (S503) is completed, the deduplication processing unit 114 performs processing according to the presence or absence of hash value registration in order from the head chunk of the write range. Specifically, the deduplication processing unit 114 determines whether it is necessary to register the hash value of the chunk to be processed by the deduplication processing unit 114 (S504).

If the hash value of the chunk to be processed is registered in the hash table 152 (S504: YES, for example, chunk 0001 in FIG. 5), the deduplication processing unit 114 determines duplicate data. In this case, the deduplication processing unit 114 acquires chunk storage location information from the hash entry 1521 of the hash value (S505). On the other hand, if the hash value of the chunk to be processed is not registered in the hash table 152 (S504: NO, for example chunk 0000 in FIG. 5), the deduplication processing unit 114 determines non-duplicate data. In this case, the hash value registration unit 1141 registers the hash value of the chunk to be processed and the chunk storage position in the new hash entry 1521 of the hash table 152 (S511). The chunk storage location can be determined by any method as long as it is an unused portion of the pool volume 14, for example.

In this embodiment, if there is a subsequent chunk in the write range (in the example of FIG. 8, the chunk subsequent to chunk 0000 is 0001), the hash value registration unit 1141 stores the hash value and hash entry position of the subsequent chunk in the write range. Register the subsequent hash value and the subsequent hash entry position of the hash entry 1521 for the hash value of the chunk. On the other hand, if there is no subsequent chunk in the write range (chunk 0008 in the example of FIG. 8), the hash value registration unit 1141 clears the subsequent hash value and the subsequent hash entry position of the hash entry 1521 related to the hash value of the chunk to be processed. (S512).

If it is determined to be non-duplicate data in the process of S504, when the information registration in the hash entry 1521 by the hash value registration unit 1141 is completed, the chunk data write unit 1142 writes the pool volume corresponding to the chunk storage position of the chunk to be processed. 14, the chunk data, which is non-overlapping data, is written (S513). After the chunk storage destination address information is determined, the chunk data write unit 1142 registers the chunk storage address information in the chunk address conversion table 151 of the management volume 15 (S506).

Then, the write command processing unit 111 determines whether or not registration of chunk storage address information in the chunk address conversion table 151 has been completed for all chunks within the write range (S507). If there is a chunk for which registration of chunk storage address information has not been completed (S507: NO), the write command processing unit 111 returns to S504 and repeats the above-described processing. On the other hand, when the registration of the chunk storage address information for all chunks is completed (S507: YES), the write command processing unit 111 reports write completion to the host 2 (S508). This completes a series of processing related to write processing.

<3.2 Read processing flow>
Read processing of the deduplication volume 13 will be described with reference to FIG.
FIG. 9 is a flowchart showing the flow of read processing for the deduplication volume 13. As shown in FIG. First, the I/O processing unit 11 of the disk array device 1 receives a read command for the deduplication volume 13 from the host 2 (S701). When the I/O processing unit 11 of the disk array device 1 receives a read command for the deduplication volume 13 from the host 2, the read command processing unit 112 secures a work memory (work buffer) for creating read data in the memory 12. (S702).

Subsequently, the read command processing unit 112 reads the chunk address conversion table 151 for all chunks in order from the first chunk within the read range (within the read address) (S703), and stores chunks from the chunk address conversion table 151. A position is acquired (S704). Then, the read command processing unit 112 reads the data of the pool volume 14 corresponding to the chunk storage position to the work memory (S705).

Subsequently, the read command processing unit 112 determines whether or not all chunks within the read range have been read (S706). If there is a chunk for which reading has not been completed (S706: NO), the read command processing unit 112 returns to S703 and repeats the above-described processing. On the other hand, when the reading of all chunks is completed (S706: YES), the read command processing unit 112 returns the read data to the host 2 and releases the work memory (S707). This completes a series of processing related to read processing.

<4. Advantage>
In a storage device that deduplicates, when checking whether write data overlaps with data already stored in the storage device, the write data is divided into chunks of a certain length, and a hash value calculated from the chunks is used. Determination is performed by comparing feature data (here, described as "hash value" for convenience) with hash values already stored in the storage. A hash table that stores hash values in storage needs to store hash values for all chunks of write data, and becomes huge. On the other hand, the hash table employs an algorithm using hierarchical management so that a desired hash value can be searched at high speed while keeping the table size as small as possible.

Therefore, in the write process, write data is divided into chunks, hash values for the chunks are calculated, and it is checked whether the calculated hash values are registered in the hash table. At this time, internal processing occurs multiple times in order to search while tracing the hierarchy of the hash table. The chunk size of storage with a high deduplication effect is, for example, 4 KB or 8 KB, and there are many cases where a write request with a size larger than the chunk size is received from the host. If the write size is larger than the chunk size, hash value searches for the number of chunks are performed, and a large amount of internal processing occurs, which may reduce the write performance.

On the other hand, in this embodiment, the storage device (for example, disk array device 1) includes a write command processing unit (for example, write command processing unit 111), a feature data registration unit (for example, hash value registration unit 1141), and a determination processing unit. (eg, hash value determination processing unit 113). The write command processing unit associates feature data (eg, hash value) corresponding to each piece of data (eg, chunk) in a predetermined management unit included in a set or series of write data. When the feature data registration unit stores the feature data corresponding to the first data (for example, the data of chunk number 0000 in FIG. 5) of the management unit included in the first write data in the storage unit (for example, the storage unit M), The feature data corresponding to the second data (for example, the data of chunk number 0001 in FIG. 5) of the management unit included in the first write data and subsequent to the first data is defined as the feature data corresponding to the first data. Associated and stored in the storage unit. When performing overlap determination on the second write data, the determination processing unit performs overlap determination using the feature data corresponding to the first data, and determines the second write data based on association with the feature data corresponding to the first data. The feature data corresponding to the data is read, and overlap determination is performed using the feature data corresponding to the second data. According to such a configuration, the load of specifying the feature data corresponding to the second data is reduced, and the write performance can be improved. For example, in the above-described embodiment, as indicated by S611 to S613 in FIG. S603) can be reduced.

In this embodiment, the feature data is a hash value calculated from the management unit data. According to such a configuration, it is possible to obtain the feature data corresponding to the data of the management unit relatively easily. As a result, the write performance can be improved.

In this embodiment, the feature data registration unit registers the feature data corresponding to the second data in the same entry as the feature data corresponding to the first data. According to such a configuration, the feature data corresponding to the first data and the feature data corresponding to the second data can be associated and stored without creating another table. As a result, the write performance can be improved while reducing the management burden as compared with the case of managing another table.

It should be noted that storing the feature data corresponding to the second data in the storage unit in association with the feature data corresponding to the first data means that the feature data corresponding to the second data is stored as the feature data corresponding to the first data. It is not limited to registering in the same entry, but includes registering information indicating the storage position of the feature data corresponding to the second data in the same entry as the feature data corresponding to the first data. In this case, the feature data corresponding to the second data can be acquired based on the information indicating the storage position of the feature data corresponding to the second data obtained based on the association.

In this embodiment, the feature data registration unit stores the feature data corresponding to the second data and the information indicating the storage location of the feature data corresponding to the second data in the same entry as the feature data corresponding to the first data. sign up. According to such a configuration, by reading the entry, it is possible to obtain the information indicating the storage position where the feature data corresponding to the second data is stored. A validity determination can be made using feature data corresponding to the data. As a result, it is possible to improve the reliability of the write.

In this embodiment, when the feature data registration unit stores feature data corresponding to the second data in the storage unit, the feature data registration unit corresponds to the third data of the management unit included in the first write data and subsequent to the second data. The feature data corresponding to the second data is stored in the storage unit in association with the feature data corresponding to the second data. For example, in the embodiment described above, the hash value corresponding to the third data is registered in the hash value entry corresponding to the second data. Then, when performing overlap determination on the second write data, the determination processing unit reads out the feature data corresponding to the third data based on the association with the feature data corresponding to the second data, and reads the feature data corresponding to the third data. Duplicate determination is performed using feature data that According to such a configuration, the feature data corresponding to the third data can be acquired based on the association with the feature data corresponding to the second data. That is, by repeating the same process, it is possible to acquire feature data corresponding to succeeding data one after another. This makes it possible to further improve the write performance.

<5. Variation>
Although the above-described embodiments relate to write data having a size larger than the chunk size, the above-described embodiments can also be applied to sequential writes in which write commands for small-sized data are consecutive, for example. The write data received by sequential write is an example of "a series of write data" and is an example of "first write data" and "second write data". When the disk array device 1 detects a sequential write from the host 2, by storing the subsequent hash value and the subsequent hash entry position in the hash entry 1521, the hash value determination in the write process to the subsequent address Hash determination (S611 to S612 in FIG. 6) can be performed using subsequent hash values. As a result, the write performance can be improved.

FIG. 10 is a diagram showing a storage device with a minimum configuration.
FIG. 11 is a diagram showing a flow of processing in a storage device with a minimum configuration. The storage device 800 includes a write command processing section 810 , a characteristic data registration section 820 and a determination processing section 830 .
The write command processing unit 810 associates each piece of data in a predetermined management unit included in a batch or a series of write data with feature data corresponding to the data (S902).
When the feature data registration unit 820 stores feature data corresponding to the first data in the management unit included in the first write data in the storage unit, the feature data registration unit 820 stores the feature data in the management unit following the first data included in the first write data. is stored in the storage unit in association with the feature data corresponding to the first data (S901).
When performing overlap determination on the second write data, the determination processing unit 830 performs overlap determination using the feature data corresponding to the first data, and performs the overlap determination based on the association with the feature data corresponding to the first data. The feature data corresponding to the second data are read out, and overlap determination is performed using the feature data corresponding to the second data (S903).

FIG. 12 is a schematic block diagram showing the computer configuration of the storage device according to at least one embodiment. Computer 5 includes CPU 6 , main memory 7 , storage 8 and interface 9 .

For example, each of the I/O processing unit 11 , memory 12 , deduplication volume 13 , pool volume 14 and management volume 15 described above is implemented in the computer 5 . The operation of the I/O processing unit 11 described above is stored in the storage 8 in the form of a program. The CPU 6 reads out the program from the storage 8, develops it in the main memory 7, and executes the above process according to the program. In addition, the CPU 6 secures storage areas in the storage 8 corresponding to the deduplication volume 13, the pool volume 14, and the management volume 15 described above according to the program. The memory 12 is implemented by the main memory 7 .

Examples of the deduplication volume 13, pool volume 14, and management volume 15 include HDD (Hard Disk Drive), SSD (Solid State Drive), magnetic disk, magneto-optical disk, CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), semiconductor memory, and the like. The deduplication volume 13, pool volume 14, and management volume 15 may be internal media directly connected to the bus of the computer 5, or external media connected to the computer 5 via the interface 9 or communication line. There may be. Further, when this program is distributed to the computer 5 through a communication line, the computer 5 that receives the distribution may develop the program in the main memory 7 and execute the above process. In at least one embodiment, storage 8 is a non-transitory, tangible storage medium.

Further, the program may implement part of the functions described above. Furthermore, the program may be a file capable of realizing the above functions in combination with a program already recorded in the computer system, that is, a so-called difference file (difference program).

While several embodiments of the invention have been described, these embodiments are examples and do not limit the scope of the invention. Various additions, omissions, replacements, and modifications may be made to these embodiments without departing from the scope of the invention.

S... Information processing system 1... Disk array device 2... Host 11... I/O processing unit 12... Memory 13... Deduplication volume 14... Pool volume 15... Management module 111... Write command processing unit 1111... Hash value calculation unit 112... Read command processing unit 113 Hash value determination processing unit 1131 Hash value determination method determination unit 1132 Hash value search unit 1133 Subsequent hash value comparison unit 114 Deduplication processing unit 1141 Hash value registration unit 1142 Chunk data write unit 800... Storage device 810... Write command processing unit 820... Characteristic data registration unit 830... Judgment processing unit

In order to achieve the above object, according to a first aspect of the present invention, a storage device associates feature data corresponding to each piece of data in a predetermined management unit included in a bundle or a series of write data. When the write command processing means and the first characteristic data corresponding to the first data of the management unit included in the first write data are stored in the storage means, the first characteristic data included in the first write data are stored in the storage means. a feature data registering means for storing second feature data, which is the feature data corresponding to the second data of the management unit subsequent to the first data, in the storage means in association with the first feature data; comparing the first feature data with the fourth feature data corresponding to the fourth data of the management unit included in the second write data when performing overlap determination on the write data; When a first duplication determination is performed, the first feature data and the fourth feature data match in the first duplication determination, and the second feature data can be read based on the association with the first feature data. , reading the second feature data based on the association with the first feature data, and reading the second feature data and the fifth data of the management unit included in the second write data and subsequent to the fourth data; determination processing means for performing a second overlap determination for comparing the fifth feature data, which is the feature data corresponding to , and determining whether or not the fourth data is duplicate data based on the result of the first overlap determination and duplicate elimination processing means for determining whether or not the fifth data is duplicate data based on the result of the second duplicate determination.

To achieve the above object, according to a second aspect of the present invention, an information processing system includes a host and a storage device communicably connected to the host. The storage device comprises: write command processing means for associating characteristic data corresponding to each piece of data in a predetermined management unit included in a set or a series of write data; When the first feature data , which is the feature data corresponding to the first data, is stored in the storage means, it corresponds to the second data of the management unit included in the first write data and subsequent to the first data. a feature data registering means for storing the second feature data , which is the feature data, in the storage means in association with the first feature data; and fourth feature data, which is the feature data corresponding to the fourth data of the management unit included in the second write data . If the data and the fourth feature data match and the second feature data can be read based on the association with the first feature data, the second feature data is read based on the association with the first feature data. reading and comparing the second feature data with the fifth feature data which is the feature data corresponding to the fifth data of the management unit included in the second write data and subsequent to the fourth data ; determination processing means for performing 2- duplication determination; determining whether or not the fourth data is duplicate data based on the result of the first duplication determination; determining whether the fifth data is duplicate data based on the result of the second duplication determination; and deduplication processing means for determining whether the data is duplicated data .

In order to achieve the above object, according to a third aspect of the present invention, an information processing method provides, for each data of a predetermined management unit included in a set or a series of write data, feature data corresponding to the data. When storing the first feature data, which is the feature data corresponding to the first data of the management unit included in the first write data, the first feature data included in the first write data and subsequent to the first data When the second feature data , which is the feature data corresponding to the second data in the management unit, is stored in association with the first feature data, and the duplication determination is performed on the second write data, the first performing a first duplication determination for comparing the feature data with the fourth feature data, which is the feature data corresponding to the fourth data of the management unit included in the second write data; When the first feature data and the fourth feature data match and the second feature data can be read based on the association with the first feature data, the second feature data can be read based on the association with the first feature data. reading data, and comparing the second feature data with the fifth feature data, which is the feature data corresponding to the fifth data of the management unit included in the second write data and subsequent to the fourth data; determining whether the fourth data is duplicate data based on the result of the first duplication determination; determining whether the fifth data is duplicate data based on the result of the second duplication determination; It includes determining whether or not it is duplicate data .

In order to achieve the above object, according to the fourth aspect of the present invention, a program is provided in a computer for each data in a predetermined management unit included in a set or a series of write data, characteristic data corresponding to the data and when storing the first feature data, which is the feature data corresponding to the first data of the management unit included in the first write data, the first data included in the first write data storing the second feature data , which is the feature data corresponding to the second data of the succeeding management unit, in association with the first feature data; and performing duplication determination on the second write data. , performing a first duplication determination for comparing the first feature data with the fourth feature data, which is the feature data corresponding to the fourth data of the management unit included in the second write data, and performing the first duplication determination ; If the first feature data and the fourth feature data match in determination and the second feature data can be read based on the association with the first feature data, based on the association with the first feature data reading out the second feature data, and a fifth feature that is the second feature data and the feature data corresponding to the fifth data of the management unit that is included in the second write data and succeeds the fourth data; performing a second overlap determination comparing the data , determining whether the fourth data is overlap data based on the result of the first overlap determination, and determining whether the fourth data is overlap data based on the result of the second overlap determination; and determining whether the fifth data is duplicate data .

Claims (9)

write command processing means for associating feature data corresponding to each piece of data in a predetermined management unit included in a set or a series of write data;
When the feature data corresponding to the first data of the management unit included in the first write data is stored in the storage means, the first data of the management unit included in the first write data and subsequent to the first data is stored in the storage means. a feature data registering means for storing the feature data corresponding to the second data in the storage means in association with the feature data corresponding to the first data;
When the overlap determination is performed on the second write data, the overlap determination is performed using the feature data corresponding to the first data, and the second write data is associated with the feature data corresponding to the first data. determination processing means for reading the feature data corresponding to the data and performing overlap determination using the feature data corresponding to the second data;
A storage device with The feature data is a hash value calculated from the data of the management unit,
The storage device according to claim 1. The feature data registration means registers the feature data corresponding to the second data in the same entry as the feature data corresponding to the first data.
3. The storage device according to claim 1 or 2. The feature data registration means registers the feature data corresponding to the second data and the information indicating the storage location of the feature data corresponding to the second data to be the same as the feature data corresponding to the first data. register for entry
The storage device according to any one of claims 1 to 3. The storage means stores table information including a plurality of hierarchies, and position information of the entry is stored in table information of the lowest layer of the plurality of hierarchies.
5. The storage device according to claim 3 or 4. When the feature data registering means stores the feature data corresponding to the second data in the storage means, the third data of the management unit included in the first write data and succeeding the second data. the feature data corresponding to is stored in the storage means in association with the feature data corresponding to the second data;
The determination processing means reads the feature data corresponding to the third data based on the association with the feature data corresponding to the second data when performing overlap determination on the second write data, and performing overlap determination using the feature data corresponding to the third data;
The storage device according to any one of claims 1 to 5. host and
a storage device communicably connected to the host;
with
The storage device is
write command processing means for associating feature data corresponding to each piece of data in a predetermined management unit included in a set or a series of write data;
When the feature data corresponding to the first data of the management unit included in the first write data is stored in the storage means, the first data of the management unit included in the first write data and subsequent to the first data is stored in the storage means. a feature data registering means for storing the feature data corresponding to the second data in the storage means in association with the feature data corresponding to the first data;
When the overlap determination is performed on the second write data, the overlap determination is performed using the feature data corresponding to the first data, and the second write data is associated with the feature data corresponding to the first data. determination processing means for reading the feature data corresponding to the data and performing overlap determination using the feature data corresponding to the second data;
An information processing system, including; Associate feature data corresponding to each data in a predetermined management unit included in a set or a series of write data,
When storing the characteristic data corresponding to the first data of the management unit included in the first write data, the second data of the management unit included in the first write data and succeeding the first data storing the corresponding feature data in association with the feature data corresponding to the first data;
When the overlap determination is performed on the second write data, the overlap determination is performed using the feature data corresponding to the first data, and the second write data is associated with the feature data corresponding to the first data. reading the feature data corresponding to the data, and performing overlap determination using the feature data corresponding to the second data;
Information processing methods. to the computer,
Associating feature data corresponding to each piece of data in a predetermined management unit included in a bundle or a series of write data;
When storing the characteristic data corresponding to the first data of the management unit included in the first write data, the second data of the management unit included in the first write data and succeeding the first data storing the corresponding feature data in association with the feature data corresponding to the first data;
When the overlap determination is performed on the second write data, the overlap determination is performed using the feature data corresponding to the first data, and the second write data is associated with the feature data corresponding to the first data. reading the feature data corresponding to the data and performing overlap determination using the feature data corresponding to the second data;
program to run.

Images