JP6888446B2 - Information processing device, deduplication rate identification method and deduplication rate identification program - Google Patents

Description

The present invention relates to an information processing device, a deduplication rate specifying method, and a deduplication rate specifying program.

In the storage device, the disk capacity is reduced by the deduplication function that prevents the same data from being stored. Deduplication is performed for each thin provisioning pool. Here, the thin provisioning pool is a physical area in which data is stored by thin provisioning. In the following description, the thin provisioning pool is simply referred to as a pool.

The storage device hashes unit data of a predetermined size using the SHA-1 function, converts it into a hash value of 20 bytes, and stores it in the main memory. Here, SHA-1 is a hash function. In addition, the storage device stores unit data on an optical disc.

When the unit data is updated, the storage device uses the SHA-1 function to hash the data into a 20-byte hash value, and searches for the same existing hash value to obtain the same data. Determine if it exists.

If the same hash value is not found, the storage device stores the hash value in the main memory and stores the updated unit data in the disk. On the other hand, if the same hash value is found, the storage device increments the reference count associated with the hash value by one. The reference count is the number in the pool of unit data corresponding to the hash value.

In addition, there is a technique for classifying texts in relation to a technique for checking the identity of data for deduplication of data. This technique performs inter-text implication recognition for a given text, selects individual texts, creates groups of text that imply the selected text, and is based on the degree of member multiplicity between groups. Integrate groups when certain conditions are met. According to this technology, multiple texts can be classified into groups whose appearance can be grasped, and texts that have semantic implications are classified into the same group even if they are not determined to have implications. can do.

Further, in relation to the allocation of storage devices to users, there is a technique for efficiently and fairly managing the storage resource usage capacity for each user in a storage system having a hierarchical structure including a plurality of storage devices. This technology calculates the ideal usage amount, which is information representing the ideal usage capacity distribution of each storage device for each user, from the storage cost coefficient and the user cost distribution information, and for each user, the performance of each storage device is calculated. The ideal usage is assigned in descending order.

Japanese Patent No. 6008067 Special Table 2012-516479

If you are operating a business using multiple pools, you may want to integrate multiple pools. However, since there is a possibility that data may be duplicated between a plurality of pools, there is a problem that the size of the physical area required after integration cannot be estimated in advance. It is possible to calculate the size of the physical area required after integration by actually reading the data of a plurality of pools, calculating the hash value, and determining the duplication, but the processing takes time.

One aspect of the present invention is to efficiently calculate the deduplication rate when a plurality of pools are integrated in order to estimate in advance the size of the physical region required after integration.

In one aspect, the information processing apparatus has a first calculation unit and a second calculation unit. The first calculation unit, for a given size unit data, calculates the total number of unit data included in the duplication number and the two virtual storage pool that indicates the number of unit data to be duplicated between two virtual storage pool .. The second calculation unit calculates the reduction rate of the physical region by deduplication using the multiples and the total number calculated by the first calculation unit, and calculates the deduplication rate from the reduction rate.

In one aspect, the present invention can efficiently calculate the deduplication rate when a plurality of pools are integrated.

FIG. 1 is a diagram showing a hardware configuration of an information processing system according to an embodiment. FIG. 2 is a diagram showing a functional configuration of a CM. FIG. 3 is a diagram showing an example of a hash table. FIG. 4 is a diagram for explaining acquisition of hash values and reference counts from other CMs. FIG. 5 is a flowchart showing the flow of the deduplication rate calculation process. FIG. 6 is a flowchart showing the flow of the multiple calculation process.

Hereinafter, examples of the information processing apparatus disclosed in the present application, the deduplication rate specifying method, and the deduplication rate specifying program will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the disclosed technology.

First, the hardware configuration of the information processing system according to the embodiment will be described. FIG. 1 is a diagram showing a hardware configuration of an information processing system according to an embodiment. As shown in FIG. 1, the information processing system 1 according to the embodiment includes a host 2 and a storage device 3. The host 2 uses the storage device 3 to perform information processing. The storage device 3 stores the data used by the host 2.

The storage device 3 has a CM (Controller Module) 4 and a volume storage device 9. The CM 4 is a control device that controls the storage device 3 and is also an information processing device that performs information processing. The volume storage device 9 stores a plurality of volumes 9a. The volume storage device 9 is, for example, a plurality of HDDs (Hard Disk Drives) and SSDs (Solid State Drives).

The CM 4 has a CA (Channel Adapter) 5, a CPU 6 (Central Processing Unit), a main memory 7, and two FC (Fiber Channel) 8.

CA5 is an interface with host 2. The CPU 6 is a central processing unit that reads a program from the main memory 7 and executes it. The main memory 7 is a RAM (Random Access Memory) that stores a program, a result during execution of the program, and the like. The FC8 is an interface with the volume storage device 9. FC8 is made redundant.

Although only one storage device 3 is shown here for convenience of explanation, the information processing system 1 may have a plurality of storage devices 3. Further, the storage device 3 may have a plurality of CM4s.

Next, the functional configuration of CM4 will be described. FIG. 2 is a diagram showing a functional configuration of CM4. As shown in FIG. 2, CM4 has a deduplication rate specifying unit 4a. The deduplication rate specifying unit 4a specifies the deduplication rate between the two pools using the following equation (1).

Here, the total physical usage is the size obtained by adding the physical usage of the two pools. The total number of reference counts is the sum of the reference counts of the two pools. The chunk size is the size of the unit data, which is the unit of deduplication, for example, 4 kilobytes (KB).

The deduplication rate is the physical usage after integration ((total physical usage)-(number of duplicate hash values) x (chunk size)) and logical usage ((total number of reference counts) x (chunk size)). It is a value based on the ratio of. For the physical usage after integration, the size of the overlapping portion between the two pools ((the number of overlapping hash values) × (chunk size)) is subtracted from (total physical usage).

The deduplication rate specifying unit 4a receives the identifiers of the two pools input by the user using, for example, a keyboard or a mouse, specifies the deduplication rate when the two pools are integrated, and sets the specified deduplication rate to 2. Output with the physical usage required when two pools are integrated. The deduplication rate specifying unit 4a includes a storage unit 40, a multiple calculation unit 41, a deduplication rate calculation unit 42, and a communication unit 43.

The storage unit 40 stores the information in the hash table. Further, the storage unit 40 includes an area for storing data temporarily used when calculating the deduplication rate, a buffer used for communication with another CM4, and the like.

FIG. 3 is a diagram showing an example of a hash table. As shown in FIG. 3, the hash table is a table in which a LUN (Logical Unit No), an LBA (Logical Block Address), and a reference count are associated with a hash value. The storage location of the unit data corresponding to the hash value in the storage device 3 is specified by the combination of LUN and LBA.

For example, the unit data having a hash value of “h” is stored at a position where the LUN of the storage device 3 is “l” and the LBA is “a”, and there are “n” of the same unit data in the pool.

The multiple calculation unit 41 calculates the number of hash values overlapping between the multiple pools, that is, the total number of reference counts, and the total number of reference counts using the hash tables of the two pools. Specifically, the multiple multiple calculation unit 41 determines whether or not the hash value of the first hash table exists in the second hash table, and if so, increments the multiple plural by 1 (1). ). Further, the multiple calculation unit 41 adds the hash value of the first hash table to the total number of reference counts (2). The multiple calculation unit 41 performs the processes (1) and (2) above for all hash values in the first hash table. Further, the multiple calculation unit 41 separately performs a process of adding all the reference counts of the second hash table to the total number of reference counts.

The deduplication rate calculation unit 42 calculates the deduplication rate using the multiples calculated by the multiple calculation unit 41 and the total number of reference counts. Then, the deduplication rate calculation unit 42 outputs the calculated deduplication rate together with the physical usage amount required when the two pools are integrated. The deduplication rate calculation unit 42 displays, for example, the physical usage amount and the deduplication rate required when the two pools are integrated on the display device of the storage device 3.

The communication unit 43 acquires the hash value and the reference count registered in the hash table from the other CM4 when the two integrated pools are controlled by different CM4s. Further, when the communication unit 43 receives a request for acquisition of the hash value and the reference count registered in the hash table from another CM4, the communication unit 43 transmits the hash value and the reference count.

FIG. 4 is a diagram for explaining acquisition of hash values and reference counts from other CM4s. FIG. 4 shows a case where CM # 1 controls pool # 1, CM # 2 controls pool # 2, and pool # 1 and pool # 2 are integrated. Further, the communication unit 43 of CM # 2 acquires the information of the hash table of pool # 1.

As shown in FIG. 4, the communication unit 43 of CM # 2 prepares a buffer for storing the hash value and reference count of pool # 1. Then, the communication unit 43 of CM # 2 issues an acquisition request with acquisition position information in order to acquire the hash value and reference count of pool # 1 (1). Here, the acquired position information is, for example, information such as q items from the p-th position in the reference table.

When the communication unit 43 of CM # 1 receives the acquisition request, it reads the hash value and the reference count from the hash table of pool # 1 based on the acquisition position information and transmits them to CM # 2 (2). Then, the communication unit 43 of CM # 2 receives the transmitted hash value and the reference count and stores them in the buffer.

The multiple calculation unit 41 of CM # 2 calculates the total number of multiples and reference counts using the hash table of pool # 2 in CM # 2, the hash value stored in the buffer, and the reference count. When all the hash values and reference counts stored in the buffer are processed, the communication unit 43 of CM # 2 adds acquisition position information and issues an acquisition request.

The deduplication rate specifying unit 4a is realized by reading from the main memory 7 shown in FIG. 1 and executing the deduplication rate specifying program having the same function by the CPU 6. The deduplication rate specifying program is stored in a CD-R (Compact Disc), which is an example of a recording medium that can be read by CM4, is read from the CD-R, and is stored in the volume storage device 9. Alternatively, the deduplication rate specifying program is stored in a database or the like of a computer system connected via a network, read from these databases, and stored in the volume storage device 9. The deduplication rate specifying program stored in the volume storage device 9 is read into the main memory 7 and executed by the CPU 6.

Next, the flow of the deduplication rate calculation process will be described. FIG. 5 is a flowchart showing the flow of the deduplication rate calculation process. As shown in FIG. 5, the deduplication rate calculation unit 42 secures various areas for temporarily storing information in the storage unit 40 (step S1).

Then, the deduplication rate calculation unit 42 requests the multiple calculation unit 41 to execute the multiple calculation process for calculating the total number of the multiple and the reference count (step S2). Then, the deduplication rate calculation unit 42 acquires the physical usage of pool # 1 and pool # 2 (step S3), and calculates the deduplication rate (step S4).

Then, the deduplication rate calculation unit 42 outputs the required physical usage amount and the deduplication removal rate (step S5), and releases various areas reserved for temporarily storing the information (step S6).

In this way, the deduplication rate calculation unit 42 calculates and outputs the deduplication rate using the multiples, the total number of reference counts, and the physical usage of pools # 1 and pool # 2, so that the user can use two pools. You can know the physical usage required after the integration of.

Next, the flow of the multiple calculation process will be described. FIG. 6 is a flowchart showing the flow of the multiple calculation process. As shown in FIG. 6, the multiple multiple calculation unit 41 initializes the multiple multiple and the total number of reference counts to 0 (step S21).

Then, the multiple calculation unit 41 acquires the hash value and the reference count at the beginning of the pool # 1 (step S22). Then, the multiple multiple calculation unit 41 determines whether or not the acquired hash value exists in the hash table of pool # 2 (step S23), and if so, increments the multiple plural (step S24).

Then, the multiple calculation unit 41 adds the reference count of pool # 1 to the total number of reference counts (step S25). Then, when the search is for the hash value at the beginning of pool # 1, the multiple calculation unit 41 adds all the reference counts of pool # 2 to the total number of reference counts (step S26).

Then, the multiple calculation unit 41 determines whether or not all the hash values of the pool # 1 have been processed (step S27), and ends the processing when all the hash values of the pool # 1 have been processed. .. On the other hand, when there is an unprocessed hash value in pool # 1, the multiple calculation unit 41 acquires the next hash value and reference count of pool # 1 (step S28), and returns to step S23.

In this way, the deduplication rate calculation unit 42 can calculate the deduplication rate by calculating the total number of the deduplication and the reference counts by the deduplication calculation unit 41.

As described above, in the embodiment, the multiple calculation unit 41 calculates the multiple multiple unit data of the two pools and the total number of reference counts of the two pools. Then, the deduplication rate calculation unit 42 calculates the deduplication rate by using the multiples and the total number of reference counts. Therefore, the deduplication rate specifying unit 4a can calculate the deduplication rate when the two pools are integrated. Therefore, the user can know the physical usage required when the two pools are integrated before the integration.

The higher the deduplication rate, the smaller the physical usage after integration, which has the advantage of integration. Comparing the result of adding the physical usage of each pool before integration with the result of physical usage after integration, if the physical usage after integration is small, there is a merit of integration.

Further, since the deduplication rate specifying unit 4a does not need to read the actual data and process it, the processing time can be reduced as compared with reading the actual data and examining the deduplication rate. Further, in the main memory 7, 8 bytes for storing the duplicates, 8 bytes × 2 for storing the physical usage of the pools # 1 and # 2, and 8 bytes for storing the total number of reference counts of the pools # 1 and # 2. It suffices to have one byte to store the deduplication rate, and the memory size can be reduced. Even when pool # 1 and pool # 2 are controlled by different CM4s, it is sufficient that there is a buffer that stores the hash value and reference count information acquired from the hash tables of other pools.

Further, in the embodiment, the deduplication rate calculation unit 42 calculates the deduplication size by multiplying the multiples by the chunk size, and subtracts the deduplication size from the total physical usage amount to calculate the size after deduplication. Further, the deduplication rate calculation unit 42 multiplies the total number of reference counts by the chunk size to calculate the total size of the area obtained by combining the two pools. Then, the deduplication rate calculation unit 42 calculates the reduction rate by dividing the size after deduplication by the total size, and subtracts the reduction rate from 1 to calculate the deduplication rate. Therefore, the deduplication rate calculation unit 42 can accurately calculate the deduplication rate.

Further, in the embodiment, when pool # 1 and pool # 2 are controlled by different CM4s, the communication unit 43 of CM4 that controls pool # 2 has a hash value and a reference registered in the hash table of pool # 1. Get the count. Then, the multiple calculation unit 41 of the CM4 that controls the pool # 2 calculates the multiple and the total number of reference counts using the hash value and the reference count acquired by the communication unit 43 and the hash table of the pool # 2. Therefore, the multiple calculation unit 41 can calculate the multiple and the total number of reference counts even when the two pools are controlled by different CM4s.

Further, in the embodiment, the case of the thin provisioning pool has been described, but the present invention is not limited to this, and can be applied to a virtual storage pool in which deduplication is performed.

Further, in the examples, the case where two pools are integrated has been described, but the present invention is not limited to this, and the present invention can be similarly applied to the case where three or more pools are integrated.

Further, in the examples, the case where the CM4 calculates the deduplication rate has been described, but the present invention is not limited to this, and the same can be applied to the case where the host 2 calculates the deduplication rate. it can.

1 Information processing system 2 Host 3 Storage device 4 CM
4a Deduplication rate identification part 5 CA
6 CPU
7 Main memory 8 FC
9 Volume storage device 9a Volume 40 Storage unit 41 Multiple multiple calculation unit 42 Deduplication rate calculation unit 43 Communication unit

Claims (6)

For a given size unit data, the first calculating unit for calculating the total number of the unit data included in the two duplication number and the two virtual storage pool that indicates the number of the unit data to be duplicated between the virtual storage pool When,
It is characterized by having a second calculation unit that calculates the reduction rate of the physical area by deduplication using the multiple and total number calculated by the first calculation unit and calculates the deduplication rate from the reduction rate. Information processing device. The second calculation unit calculates the overlapping size of the overlapping area by multiplying the multiple by the predetermined size, and subtracts the overlapping size from the sum of the sizes of the physical areas used by the two virtual storage pools. The size after deduplication is calculated, the total size is multiplied by the predetermined size to calculate the total size of the combined area of the two virtual storage pools, and the reduction rate obtained by dividing the size after deduplication by the total size. The information processing apparatus according to claim 1, wherein the value obtained by subtracting 1 from 1 is calculated as the deduplication rate. The first calculation unit calculates the multiple and the total number by using the hash correspondence information in which the number of unit data for which the hash value is calculated by the hash calculation in the virtual storage pool and the hash value are associated with each other. The information processing apparatus according to claim 1 or 2. One of the two virtual storage pools is controlled by the other information processing device.
Information about the virtual storage pool controlled by itself is registered in the hash correspondence information, and the information is registered.
It has a receiving unit that receives information registered in the hash correspondence information of the other information processing device from the other control device.
The first calculation unit is characterized in that the multiple and the total number are calculated by using the information registered in the hash correspondence information of itself and the information acquired by the receiving unit from another storage control device. The information processing device according to claim 3. The computer
For a given size unit data, and calculates the total number of the unit data included in the two duplication number and the two virtual storage pool that indicates the number of the unit data to be duplicated between the virtual storage pool,
A method for specifying a deduplication rate, which comprises calculating the reduction rate of a physical area by deduplication using the calculated multiples and the total number, and executing a process of calculating the deduplication rate from the reduction rate. On the computer
For a given size unit data, and calculates the total number of the unit data included in the two duplication number and the two virtual storage pool that indicates the number of the unit data to be duplicated between the virtual storage pool,
A deduplication rate identification program characterized by calculating the reduction rate of a physical area due to deduplication using the calculated multiples and the total number, and executing a process of calculating the deduplication rate from the reduction rate.

Images