JP2019016293A - Information processing device, deduplication rate determination method and deduplication rate determination program - Google Patents

Abstract

To provide an information processing device capable of calculating the deduplication rate when multiple pools are integrated for allowing a user to estimate in advance the size of a physical area required after the integration.SOLUTION: A deduplication rate determination part 4a of a CM (Controller Module) 4 comprises: a duplication number calculation part 41; and a deduplication rate calculation part 42. The duplication number calculation part 41 is configured to calculate the deduplication number of unit data in two pools and the total number of reference count of the two pools. The deduplication rate calculation part 42 is configured to calculate the deduplication rate by using the deduplication number and the total number of the reference count.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing apparatus, a duplicate removal rate specifying method, and a duplicate removal rate specifying program.

  In the storage apparatus, the disk capacity is reduced by a deduplication function that prevents the same data from being stored. Deduplication is performed for each thin provisioning pool. Here, the thin provisioning pool is a physical area in which data is stored by thin provisioning. In the following description, the thin provisioning pool is simply referred to as a pool.

  The storage apparatus hashes the unit data of a predetermined size using the SHA-1 function, and stores it as a 20-byte hash value on the main memory. Here, SHA-1 is a hash function. The storage device also stores unit data on the disk.

  When the unit data is updated, the storage apparatus performs hashing using the SHA-1 function, makes a hash value of 20 bytes, and searches for the same hash value as the existing hash value. It is determined whether or not there is.

  If the same hash value is not found, the storage device stores the hash value on the main memory, and stores the updated unit data on the disk. On the other hand, if the same hash value is found, the storage apparatus increments the reference count associated with the hash value by one. The reference count is the number in the pool of unit data corresponding to the hash value.

  In addition, there is a technique for classifying texts in connection with a technique for checking the identity of data in order to eliminate duplication of data. This technology recognizes implications between texts for a given text, selects individual texts, generates a group whose text entails the selected text, and based on the degree of member overlap between the groups. Merge groups when certain conditions are met. According to this technology, it is possible to classify multiple texts into groups that allow an overview, and classify texts that have semantic implications into the same group even if they are not determined to have implications. can do.

  In addition, there is a technique for efficiently and fairly managing the storage resource usage capacity for each user in a storage system having a hierarchical structure including a plurality of storage devices in relation to the allocation of storage devices to users. This technology calculates the ideal usage amount, which is information representing the ideal usage capacity distribution of each storage device for each user, from the storage cost coefficient and user cost distribution information. The ideal usage is allocated in order from the highest.

Japanese Patent No. 6008067 Special table 2012-516479 gazette

  When business operations are performed using multiple pools, it may be desirable to integrate multiple pools. However, there is a possibility that data may be duplicated between a plurality of pools, and thus there is a problem that the size of a physical area required after integration cannot be estimated in advance. Note that it is possible to calculate the size of the physical area required after integration by actually reading data from a plurality of pools, calculating hash values, and determining duplication, but processing takes time.

  In one aspect, an object of the present invention is to efficiently calculate a deduplication rate when a plurality of pools are integrated in order to estimate in advance the size of a physical area required after integration.

  In one aspect, the information processing apparatus includes a first calculation unit and a second calculation unit. The first calculation unit calculates the overlap between two virtual storage pools and the total number of unit data included in the two virtual storage pools for unit data of a predetermined size. The second calculation unit calculates an overlap removal rate using the overlap number and the total number calculated by the first calculation unit.

  In one aspect, the present invention can efficiently calculate the deduplication rate when a plurality of pools are integrated.

FIG. 1 is a diagram illustrating a hardware configuration of the information processing system according to the embodiment. FIG. 2 is a diagram illustrating a functional configuration of the CM. FIG. 3 is a diagram illustrating an example of a hash table. FIG. 4 is a diagram for explaining the acquisition of hash values and reference counts from other CMs. FIG. 5 is a flowchart showing the flow of the deduplication rate calculation process. FIG. 6 is a flowchart showing a flow of the duplication number calculation process.

  Embodiments of an information processing apparatus, a duplicate removal rate specifying method, and a duplicate removal rate specifying program disclosed in the present application will be described below in detail with reference to the drawings. Note that this embodiment does not limit the disclosed technology.

  First, the hardware configuration of the information processing system according to the embodiment will be described. FIG. 1 is a diagram illustrating a hardware configuration of the information processing system according to the embodiment. As illustrated in FIG. 1, the information processing system 1 according to the embodiment includes a host 2 and a storage device 3. The host 2 performs information processing using the storage device 3. The storage device 3 stores data used by the host 2.

  The storage device 3 has a CM (Controller Module) 4 and a volume storage device 9. The CM 4 is a control device that controls the storage device 3 and an information processing device that performs information processing. The volume storage device 9 stores a plurality of volumes 9a. The volume storage device 9 is, for example, a plurality of HDDs (Hard Disk Drives) and SSDs (Solid State Drives).

  The CM 4 includes a CA (Channel Adapter) 5, a CPU 6 (Central Processing Unit), a main memory 7, and two FC (Fiber Channel) 8.

  The CA 5 is an interface with the host 2. The CPU 6 is a central processing unit that reads a program from the main memory 7 and executes it. The main memory 7 is a RAM (Random Access Memory) that stores programs, results of program execution, and the like. The FC 8 is an interface with the volume storage device 9. FC8 is made redundant.

  Although only one storage device 3 is shown here for convenience of explanation, the information processing system 1 may include a plurality of storage devices 3. Further, the storage device 3 may have a plurality of CMs 4.

Next, the functional configuration of CM4 will be described. FIG. 2 is a diagram showing a functional configuration of CM4. As illustrated in FIG. 2, the CM 4 includes an overlap removal rate specifying unit 4a. The deduplication rate specifying unit 4a specifies the deduplication rate between the two pools using the following equation (1).

  Here, the total physical usage is a size obtained by adding the physical usage of the two pools. The total number of reference counts is the number obtained by adding the reference counts of the two pools. The chunk size is the size of unit data that is a unit of deduplication, and is, for example, 4 kilobytes (KB).

  Deduplication rate is the combined physical usage ((total physical usage)-(number of duplicate hash values) x (chunk size)) and logical usage ((total number of reference counts) x (chunk size)) It is a value based on the ratio of. The physical usage after integration is subtracted from (total physical usage) the size of the overlapping portion between the two pools ((number of overlapping hash values) × (chunk size)).

  The deduplication rate specifying unit 4a receives the identifiers of two pools input by the user using, for example, a keyboard or a mouse, specifies the deduplication rate when the two pools are integrated, and sets the specified deduplication rate to 2 Output together with the physical usage required when two pools are combined. The duplicate removal rate specifying unit 4 a includes a storage unit 40, a duplicate number calculation unit 41, a duplicate removal rate calculation unit 42, and a communication unit 43.

  The storage unit 40 stores hash table information. Further, the storage unit 40 includes an area for storing data temporarily used when calculating the deduplication rate, a buffer used for communication with other CMs 4 and the like.

  FIG. 3 is a diagram illustrating an example of a hash table. As shown in FIG. 3, the hash table is a table that associates a LUN (Logical Unit No), an LBA (Logical Block Address), and a reference count with a hash value. The storage location of the unit data corresponding to the hash value in the storage apparatus 3 is specified by a combination of LUN and LBA.

  For example, unit data whose hash value is “h” is stored at a location where the LUN of the storage apparatus 3 is “1” and LBA is “a”, and there are “n” same unit data in the pool.

  The duplication number calculation unit 41 calculates the duplication number, that is, the number of hash values that overlap between two pools and the total number of reference counts using the hash tables of the two pools. Specifically, the duplication number calculation unit 41 determines whether or not the hash value of the first hash table exists in the second hash table, and if it exists, the duplication number is incremented by 1 (1 ). Also, the duplication number calculation unit 41 adds the hash value of the first hash table to the total number of reference counts (2). The duplicate calculation unit 41 performs the processes (1) and (2) on all the hash values in the first hash table. In addition, the duplication number calculation unit 41 separately performs a process of adding all the reference counts of the second hash table to the total number of reference counts.

  The duplication removal rate calculation unit 42 calculates the duplication removal rate using the duplication number calculated by the duplication number calculation unit 41 and the total number of reference counts. Then, the deduplication rate calculation unit 42 outputs the calculated deduplication rate together with the physical usage required when the two pools are integrated. The deduplication rate calculation unit 42 displays, for example, the physical usage amount and deduplication rate required when two pools are integrated on the display device of the storage apparatus 3.

  When the two pools to be integrated are controlled by different CMs 4, the communication unit 43 acquires a hash value and a reference count registered in the hash table from the other CMs 4. When the communication unit 43 receives a hash value and reference count acquisition request registered in the hash table from another CM 4, the communication unit 43 transmits the hash value and the reference count.

  FIG. 4 is a diagram for explaining acquisition of a hash value and a reference count from another CM 4. FIG. 4 shows a case where CM # 1 controls pool # 1, CM # 2 controls pool # 2, and pool # 1 and pool # 2 are integrated. In addition, the communication unit 43 of CM # 2 acquires information on the hash table of pool # 1.

  As shown in FIG. 4, the communication unit 43 of CM # 2 prepares a buffer for storing the hash value and reference count of pool # 1. Then, the communication unit 43 of the CM # 2 issues an acquisition request with the acquisition position information added thereto in order to acquire the hash value and the reference count of the pool # 1 (1). Here, the acquisition position information is, for example, information from p-th to q-th reference table.

  When receiving the acquisition request, the CM # 1 communication unit 43 reads the hash value and the reference count from the hash table of the pool # 1 based on the acquisition position information, and transmits the hash value and the reference count to the CM # 2 (2). Then, the communication unit 43 of CM # 2 receives the transmitted hash value and reference count and stores them in the buffer.

  The duplication number calculation unit 41 of CM # 2 calculates the total number of duplication numbers and reference counts using the hash value and reference count stored in the buffer of the pool # 2 in CM # 2 and the buffer. When all the hash values and the reference count stored in the buffer are processed, the communication unit 43 of CM # 2 issues an acquisition request with the acquisition position information.

  The deduplication rate specifying unit 4a is realized by reading a deduplication rate specifying program having the same function from the main memory 7 shown in FIG. The duplicate removal rate specifying program is stored in a CD-R (Compact Disc) which is an example of a recording medium readable by the CM 4, read from the CD-R, and stored in the volume storage device 9. Alternatively, the deduplication rate specifying program is stored in a database or the like of a computer system connected via a network, read from these databases, and stored in the volume storage device 9. The deduplication rate specifying program stored in the volume storage device 9 is read into the main memory 7 and executed by the CPU 6.

  Next, the flow of the duplicate removal rate calculation process will be described. FIG. 5 is a flowchart showing the flow of the deduplication rate calculation process. As shown in FIG. 5, the deduplication rate calculation unit 42 secures various areas for temporarily storing information in the storage unit 40 (step S1).

  Then, the duplication removal rate calculation unit 42 requests the duplication number calculation unit 41 to execute duplication number calculation processing for calculating the total number of duplication numbers and reference counts (step S2). Then, the duplicate removal rate calculation unit 42 acquires the physical usage amounts of the pool # 1 and the pool # 2 (step S3), and calculates the duplicate removal rate (step S4).

  The deduplication rate calculation unit 42 then outputs the necessary physical usage and deduplication rate (step S5), and releases various areas reserved for temporarily storing information (step S6).

  In this way, since the deduplication rate calculation unit 42 calculates and outputs the deduplication rate using the duplication number, the total number of reference counts, and the physical usage of the pool # 1 and the pool # 2, the user can create two pools You can know the physical usage required after integration.

  Next, the flow of the overlapping number calculation process will be described. FIG. 6 is a flowchart showing a flow of the duplication number calculation process. As shown in FIG. 6, the duplication number calculation unit 41 initializes the duplication number and the total number of reference counts to 0 (step S21).

  Then, the duplication number calculation unit 41 acquires the first hash value and the reference count of the pool # 1 (step S22). Then, the duplication number calculation unit 41 determines whether or not the acquired hash value exists in the hash table of the pool # 2 (step S23), and if it exists, increments the duplication number (step S24).

  Then, the duplication number calculation unit 41 adds the reference count of the pool # 1 to the total number of reference counts (step S25). Then, when the search is for the first hash value of the pool # 1, the duplication number calculation unit 41 adds the total reference count of the pool # 2 to the total number of reference counts (step S26).

  Then, the duplication number calculation unit 41 determines whether or not all the hash values of the pool # 1 have been processed (step S27), and when all the hash values of the pool # 1 have been processed, the process ends. . On the other hand, when there is an unprocessed hash value in the pool # 1, the duplication number calculation unit 41 acquires the next hash value and the reference count of the pool # 1 (step S28), and returns to step S23.

  In this way, the duplication number calculating unit 41 calculates the total number of duplication numbers and reference counts, so that the duplication removal rate calculation unit 42 can calculate the duplication removal rate.

  As described above, in the embodiment, the duplication number calculation unit 41 calculates the duplication number of unit data of two pools and the total number of reference counts of the two pools. Then, the deduplication rate calculation unit 42 calculates the deduplication rate using the duplication number and the total number of reference counts. Therefore, the duplicate removal rate specifying unit 4a can calculate the duplicate removal rate when the two pools are integrated. For this reason, the user can know the physical usage required when two pools are integrated before the integration.

  As the deduplication rate is higher, the physical usage after integration is reduced, so there is an advantage of integration. The result of adding the physical usage of each pool before the integration and the result of the physical usage after the integration are compared. If the physical usage after the integration is small, there is an advantage of integration.

  Further, since the deduplication rate specifying unit 4a does not need to read and process the actual data, the processing time can be reduced as compared with reading the actual data and checking the deduplication rate. The main memory 7 has 8 bytes for storing the duplication number, 8 bytes for storing the physical usage of the pools # 1 and # 2, and 8 bytes for storing the total number of reference counts for the pools # 1 and # 2. There is only one byte for storing the deduplication rate, and the memory size can be reduced. Even when the pool # 1 and the pool # 2 are controlled by different CMs 4, it is only necessary to have a buffer for storing the hash value acquired from the hash table of the other pool and the reference count information.

  In the embodiment, the deduplication rate calculation unit 42 calculates the overlap size by multiplying the overlap number by the chunk size, and calculates the post-deduplication size by subtracting the overlap size from the total physical usage. Further, the duplicate removal rate calculation unit 42 calculates the total size of the area obtained by combining the two pools by multiplying the total number of reference counts by the chunk size. Then, the deduplication rate calculation unit 42 calculates the reduction rate by dividing the size after deduplication by the total size, and subtracts the reduction rate from 1 to calculate the deduplication rate. Therefore, the deduplication rate calculation unit 42 can accurately calculate the deduplication rate.

  In the embodiment, when the pool # 1 and the pool # 2 are controlled by different CM4, the CM4 communication unit 43 that controls the pool # 2 uses the hash value and the reference registered in the hash table of the pool # 1. Get the count. Then, the duplication number calculation unit 41 of the CM 4 that controls the pool # 2 calculates the duplication number and the total number of reference counts using the hash value, the reference count, and the hash table of the pool # 2 acquired by the communication unit 43. Therefore, the duplication number calculation unit 41 can calculate the duplication number and the total number of reference counts even when the two pools are controlled by different CMs 4.

  In the embodiment, the case of the thin provisioning pool has been described. However, the present invention is not limited to this, and can be applied to a virtual storage pool in which deduplication is performed.

  Moreover, although the case where two pools were integrated was demonstrated in the Example, this invention is not limited to this, It can apply similarly when integrating three or more pools.

  In the embodiment, the case where the CM 4 calculates the deduplication rate has been described. However, the present invention is not limited to this, and the same applies to the case where the host 2 calculates the deduplication rate. it can.

1 Information processing system 2 Host 3 Storage device 4 CM
4a Deduplication rate specifying part 5 CA
6 CPU
7 Main memory 8 FC
9 Volume storage device 9a Volume 40 Storage unit 41 Duplicate multiple calculation unit 42 Deduplication rate calculation unit 43 Communication unit

Claims (6)

A first calculation unit for calculating a duplication number between two virtual storage pools and a total number of the unit data included in the two virtual storage pools for unit data of a predetermined size;
An information processing apparatus comprising: a second calculation unit that calculates a duplication removal rate using the duplication number and the total number calculated by the first calculation unit.   The second calculation unit calculates the overlap size of the overlap region by multiplying the overlap number by the predetermined size, and subtracts the overlap size from the sum of the sizes of the physical regions used by the two virtual storage pools A reduction ratio obtained by calculating the size after deduplication, calculating the total size of the two virtual storage pools by multiplying the total number by the predetermined size, and dividing the size after deduplication by the total size The information processing apparatus according to claim 1, wherein a value obtained by subtracting 1 from 1 is calculated as a deduplication rate.   The first calculation unit calculates the duplication number and the total number using hash correspondence information in which a hash value is associated with the number of unit data for which a hash value is calculated by hash calculation in the virtual storage pool. The information processing apparatus according to claim 1 or 2. One of the two virtual storage pools is controlled by another information processing apparatus,
In the hash correspondence information, information about the virtual storage pool controlled by itself is registered,
A receiver that receives information registered in the hash correspondence information of the other information processing apparatus from the other control apparatus;
The first calculation unit calculates the duplication number and the total number using information registered in its own hash correspondence information and information acquired by the reception unit from another storage control device. The information processing apparatus according to claim 3. Computer
Calculating a duplication number between two virtual storage pools and a total number of the unit data included in the two virtual storage pools for unit data of a predetermined size;
A deduplication rate specifying method, comprising: performing a process of calculating a deduplication rate using the calculated duplication number and the total number. On the computer,
Calculating a duplication number between two virtual storage pools and a total number of the unit data included in the two virtual storage pools for unit data of a predetermined size;
A deduplication rate identifying program characterized by causing a process to calculate a deduplication rate using the calculated duplication number and total number.

Images