JP6201340B2 - Replication system - Google Patents

Description

  The present invention relates to a replication system, and more particularly to a replication system including a storage system having a function of eliminating duplicate storage.

  In recent years, with the development and spread of computers, various types of information have been converted into digital data. As a device for storing such digital data, there are storage devices such as a magnetic tape and a magnetic disk. Since the data to be stored increases day by day and becomes enormous, a large-capacity storage system is required. In addition, reliability is required while reducing the cost of the storage device. In addition to this, it is necessary that data can be easily retrieved later. As a result, there is a demand for a storage system that can automatically increase storage capacity and performance, eliminate duplicate storage, reduce storage costs, and have high redundancy.

  In response to such a situation, in recent years, a content address storage system has been developed as shown in Patent Document 1. In this content address storage system, data is distributed and stored in a plurality of storage devices, and the storage location where the data is stored is specified by a unique content address specified according to the content of the data. Further, in the content address storage system, predetermined data is divided into a plurality of fragments, and a fragment that becomes redundant data is further added, and the plurality of fragments are stored in a plurality of storage devices, respectively. There are also things.

  Then, in the content address storage system as described above, by designating the content address later, the data stored in the storage location specified by the content address, that is, the fragment is read out, and a plurality of fragments before the division are read. Predetermined data can be restored.

  The content address is generated based on a value generated to be unique according to the data content, for example, a hash value of the data. For this reason, if it is duplicate data, the data of the same content can be acquired by referring to the data at the same storage position. Therefore, there is no need to store duplicate data separately, and duplicate recording can be eliminated to reduce the data capacity.

  In particular, in a storage system having a function of eliminating duplicate storage as described above, data to be written such as a file is divided into a plurality of block data of a predetermined capacity, compressed, and written to a storage device. In this way, the data capacity is reduced by eliminating duplicate storage in units of block data obtained by dividing the file. When the storage system is used for data backup, it is possible to save backup capacity and save bandwidth during replication. Such replication is to transmit data existing in a certain system to another system via a network or the like for the purpose of data maintenance or the like, and to replicate the data.

  In the above-described replication in the deduplication storage system, whether the data already exists in the replica system before data is transferred from the replication source system (hereinafter referred to as the master system) to the replication destination system (hereinafter referred to as the replica system). Check if. Data that already exists in the replica system can be omitted from the master system, and the amount of transferred data can be reduced. Specifically, the master system sends a digest (hash value) of the data to be transferred to the replica system, and based on the digest, the replica system checks whether the same data exists in its own system. Do. This is realized by requesting transmission of only non-existing data to the master system.

Here, the data that exists in the master system and is the replication target can be classified into the following four types.
A: At the time of writing in the master system, duplication is eliminated because it already exists in the master system, data is not actually written, and the same data also exists in the replica system.
B: At the time of writing in the master system, data was actually written because it did not exist in the master system, but the same data exists in the replica system.
C: At the time of writing in the master system, duplication is eliminated because it already exists in the master system, and data is not actually written, but the same data does not exist in the replica system.
D: At the time of writing in the master system, data was actually written because it did not exist in the master system, and the same data does not exist in the replica system.

  The numbers that the master system can know before starting replication are only the total of replication target data (A + B + C + D) and the total of the data newly written without deduplication (B + D).

  The data classified into A and B is not transferred on the network, and only the existence confirmation in the replica system is performed, so that the data is processed at a very high speed. However, since the data classified into C and D needs to be read in the master system, transferred in the network, and written in the replica system after the existence check in the replica system, the processing is very slow. Become. That is, the processing time for replication is almost occupied by the time for processing data classified as C and D.

  Replication processing is often very time consuming, especially when the network bandwidth between the master system and the replica system is narrow. For this reason, there is a high need to know a progress rate that increases linearly from the start to the end of replication and is easy to predict the required time to completion. For example, Patent Document 2 describes that a data transfer time is predicted when data is replicated.

JP 2005-235171 A JP 2012-247981 A

  Here, in order to calculate the replication progress rate, it is necessary to know the total amount of data classified into C and D for the reasons described above. However, it is difficult for a master system having data to be replicated to know this without making an inquiry in advance to the replica system. That is, it is difficult for the master system to know the total amount of data before replication is completed.

  On the other hand, the master system can obtain only “A + B + C + D” and “B + D” before executing replication. For this reason, until now, instead of “C + D”, it is assumed that “B + D” is the total amount of replication processing, and at a certain point of replication, “data transfer amount from the start of replication” is divided by “B + D”. It was considered to calculate the progress rate.

However, when the difference between “B + D” and “C + D” is large, the following problem occurs.
When (B + D)> (C + D): The progress rate is a low number and does not increase easily, and replication is completed suddenly.
When (B + D) <(C + D): The progress rate increases to almost 100% during the execution of replication.

  In either case, the prediction of the time required to complete the replication based on the progress rate at a certain point will be significantly different. From the above, there is a problem that it is difficult to calculate the replication progress rate with a value closer to the actual situation in the deduplication storage system.

  Therefore, an object of the present invention is to provide a replication system that can solve the above-mentioned problem, that is, it is difficult to calculate the replication progress rate with a value closer to the actual situation in the deduplication storage system. It is to provide.

A replication system according to one aspect of the present invention is:
The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In this case, it has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data,
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication point;
The replication execution unit investigates whether or not data to be replicated from the master side storage system to the replica side storage system is deduplicated and stored in the storage device of the replica side storage system, and performs replication. Transfer data stored in the replica storage system without being deduplicated from the master storage system to the replica storage system,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the investigation result of the data to be replicated and is not deduplicated in the storage device of the replica side storage system. Estimating the total data amount of newly stored data, and based on the estimated total data amount and the data amount of data transferred from the master storage system to the replica storage system at a predetermined replication point , Calculate the replication progress rate at the given replication point,
The configuration is as follows.

The program which is the other form of this invention is:
The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In the case of providing a function for eliminating duplicate storage by referring to data already stored in the storage device as other data, between the master-side storage system and the replica-side storage system. In the information processing device that controls replication in
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication time point, and
The replication execution unit investigates whether or not data to be replicated from the master side storage system to the replica side storage system is deduplicated and stored in the storage device of the replica side storage system, and performs replication. Transfer data stored in the replica storage system without being deduplicated from the master storage system to the replica storage system,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the investigation result of the data to be replicated and is not deduplicated in the storage device of the replica side storage system. Estimating the total data amount of newly stored data, and based on the estimated total data amount and the data amount of data transferred from the master storage system to the replica storage system at a predetermined replication point , Calculate the replication progress rate at the given replication point,
It is a program for realizing this.

Another aspect of the present invention is a replication method,
The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device Replication function between the master side storage system and the replica side storage system, which has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data. A method,
A replication execution step of replicating the data stored in the storage device of the master side storage system to the storage device of the replica side storage system;
A progress rate calculating step for calculating the progress rate of the replication at a predetermined replication point;
The replication execution step investigates whether data to be replicated from the master side storage system to the replica side storage system is stored in the storage device of the replica side storage system in a deduplicated manner and performs the replication. Transfer data stored in the replica storage system without being deduplicated from the master storage system to the replica storage system,
The progress rate calculation step is based on the investigation result of the data to be replicated, and is transferred from the master side storage system to the replica side storage system in the entire replication and is not deduplicated in the storage device of the replica side storage system. Estimating the total data amount of newly stored data, and based on the estimated total data amount and the data amount of data transferred from the master storage system to the replica storage system at a predetermined replication point , Calculate the replication progress rate at the given replication point,
The configuration is as follows.

  With the configuration as described above, the present invention can calculate the replication progress rate with a value closer to the actual condition in the storage system.

It is a block diagram which shows the structure of the replication system in Embodiment 1 of this invention. It is a block diagram which shows the outline of a structure of the master system disclosed in FIG. 1, or a replica system. It is explanatory drawing for demonstrating the mode of the data writing process in the master system or replica system disclosed in FIG. It is explanatory drawing explaining the mode of the master system or replica system data write processing disclosed in FIG. It is a figure which shows the structure of the metadata stored in the master system disclosed in FIG. It is a figure which shows the classification | category of the data replicated with the replication system disclosed in FIG. 3 is a flowchart showing an operation of file writing processing by the replication system disclosed in FIG. 1. 3 is a flowchart showing an operation of replication processing by the replication system disclosed in FIG. 1. 2 is a flowchart showing an operation of progress rate calculation processing by the replication system disclosed in FIG. 1. It is a block diagram which shows the structure of the storage system in attachment 1 of this invention.

<Embodiment 1>
A first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing the configuration of the entire replication system. FIG. 2 is a block diagram showing an outline of a storage system constituting the replication system. 3 to 6 are explanatory diagrams for explaining the operation of data write processing and replication processing in the replication system. 7 to 9 are flowcharts showing the operation of the replication system.

  Here, the present embodiment shows a specific example of the master-side storage system and the replica-side storage system constituting the replication system described in the appendix to be described later. In the following, a case will be described in which each storage system is configured by connecting a plurality of server computers. However, each storage system constituting the replication system in the present invention is not limited to being constituted by a plurality of computers, and may be constituted by one computer.

[Constitution]
As shown in FIG. 1, in the replication system of this embodiment, a master system 10 and a replica system 20 that are storage systems are connected via a network. A user system 30 is connected to the master system 10.

  First, the user system 30 reads / writes data from / to the master system 10 through general file / directory operations via the file system view providing unit 14 on the master system 10 described later. In addition, the user system 30 requests the master system 10 for a progress rate of replication, which will be described later, and browses the progress rate calculated accordingly.

  The master system 10 includes an arithmetic device (not shown) and a storage device 11, and is constructed by incorporating a program into the arithmetic device, and a master-side replication execution unit 12 and a data classification information management unit 13 And a file system view providing unit 14 and a progress information browsing unit 15. The replica system 20 includes an arithmetic device (not shown) and a storage device 21, and includes a replica-side replication execution unit 22 constructed by incorporating a program into the arithmetic device.

  The file system view providing unit 14 performs a process of writing data to the storage device 11 and a process of reading data from the storage device 11 in response to a request from the user system 30, and provides a file system view to the user system 30. . Here, the master system 10 has a function of eliminating duplicate storage when data is written to the storage device 11. The replica system 20 stores the data replicated from the master system 10 in the storage device 21, and the replica system 20 also has a function of eliminating duplicate storage. Hereinafter, the function of eliminating the duplicate storage included in the storage system 1 that is the master system 10 and the replica system 20 will be described with reference to FIGS.

  As shown in FIG. 2, the storage system 1 that is the master system 10 and the replica system 20 in the present embodiment has a configuration in which a plurality of server computers are connected. Specifically, the storage system 1 includes an accelerator node 2 that is a server computer that controls storage and reproduction operations in the storage system 1 itself, and a storage node 3 that is a server computer including a storage device that stores data. Yes. The number of accelerator nodes 2 and the number of storage nodes 3 are not limited to those shown in FIG. 2, and more nodes 2 and 3 may be connected.

  Furthermore, the storage system 1 according to the present embodiment divides and redundantly stores the data, distributes and stores the data in a plurality of storage devices, and stores the data by a unique content address set according to the content of the stored data. It is a content address storage system for specifying a stored location.

  Hereinafter, assuming that the storage system 1 is one system, the configuration and functions of the storage system 1 will be described. That is, the configuration and function of the storage system 1 described below may be provided in either the accelerator node 2 or the storage node 3. As shown in FIG. 2, the storage system 1 is not necessarily limited to including the accelerator node 2 and the storage node 3, and may have any configuration, for example, a single computer. May be. Furthermore, the storage system 1 is not limited to being a content address storage system, and may be a storage system having a deduplication function.

  Next, data write processing and read processing using content addresses by the storage system 1 in this embodiment will be described with reference to FIG. 3 and FIG.

  First, as indicated by an arrow Y1 in FIGS. 3 and 4, the storage system 1 receives an input of a file A requested to be written. Then, as shown by an arrow Y2 in FIGS. 3 and 4, the file A is divided into block data D having a predetermined capacity (for example, 64 KB).

  Subsequently, based on the data content of the divided block data D, a unique hash value H representing the data content is calculated (arrow Y3 in FIG. 4). For example, the hash value H is calculated from the data content of the block data D using a preset hash function.

  Subsequently, using the hash value H of the block data D of the file A, it is checked whether or not the block data D is already stored. Specifically, first, the block data D that has already been stored has its hash value H and a content address CA representing the storage position associated with each other and registered in an MFI (Main Fragment Index) file. Therefore, if the hash value H of the block data D calculated before storage exists in the MFI file, it can be determined that the same block data D has already been stored (arrow Y4 in FIG. 4). In this case, the content address CA associated with the hash value H in the MFI that matches the hash value H of the block data D before storage is acquired from the MFI file. Then, this content address CA is returned as the content address CA of the block data D requested to be written.

  The already stored data referred to by the returned content address CA is used as the block data D requested to be written. That is, it is assumed that the block data D requested to be written is stored by designating an area referred to by the returned content address CA as a storage destination of the block data D requested to be written. This eliminates the need to actually store the block data D relating to the write request in the storage device.

  If it is determined that the block data D related to the write request is not yet stored, the block data D related to the write request is written as follows. First, the block data D related to the write request is compressed and divided into a plurality of fragment data of a predetermined capacity as indicated by an arrow Y5 in FIG. For example, as shown by symbols D1 to D9 in FIG. 3, the data is divided into nine fragment data (divided data 41). Further, even if some of the divided fragment data is missing, redundant data is generated so that the original block data can be restored and added to the divided fragment data 41. For example, three pieces of fragment data (redundant data 42) are added as indicated by reference numerals D10 to D12 in FIG. As a result, a data set 40 composed of twelve fragment data composed of nine divided data 41 and three redundant data 42 is generated.

  Subsequently, each fragment data constituting the data set generated as described above is distributed and stored in each storage area formed in the storage device. For example, as shown in FIG. 3, when 12 pieces of fragment data D1 to D12 are generated, each piece of fragment data D1 to D12 is stored in a data storage file formed in each of a plurality of storage devices. (See arrow Y6 in FIG. 4).

  Subsequently, the storage system 1 generates a content address CA indicating the storage position of the fragment data D1 to D12 stored as described above, that is, the storage position of the block data D restored by the fragment data D1 to D12. Manage. Specifically, a part of the hash value H (short hash) calculated based on the contents of the stored block data D (for example, the top 8B (bytes) of the hash value H), information indicating the logical storage position, Are combined to generate a content address CA. Then, this content address CA is returned to the file system in the storage system 1 (arrow Y7 in FIG. 4). Then, the storage system 1 manages identification information such as the file name of the backup target data and the content address CA in association with the file system.

  Further, each storage node 3 manages the content address CA of the block data D and the hash value H of the block data D by using the MFI file. As described above, the content address CA is stored in the storage device of the accelerator node 2 or the storage node 3 in association with the information specifying the file, the hash value H, or the like.

  Furthermore, the storage system 1 performs control to read out the stored file as described above. For example, when a read request is made by designating a specific file to the storage system 1, first, based on the file system, a short hash and a logical position that are part of a hash value corresponding to the file for the read request A content address CA consisting of the above information is designated. Then, it is checked whether or not the content address CA is registered in the MFI file. If it is not registered, the requested data is not stored and an error is returned.

  On the other hand, when the content address CA related to the read request is registered, the storage location specified by the content address CA is specified, and each fragment data stored in the specified storage location is read. Read as requested data. At this time, if the data storage file in which each fragment is stored and the storage position of one fragment data in the data storage file are known, the storage position of other fragment data can be specified from the same storage position. .

  Then, the block data D is restored from each fragment data read in response to the read request. Further, a plurality of restored block data D are concatenated, restored to a group of data such as file A, and returned.

  Next, as described above, the data written in the storage device 11 of the master system 10 (master side storage system) having the deduplication function is stored in the replica system 20 (replica side storage system) also having the deduplication function. A configuration for calculating the progress rate when replicating to the apparatus 21 will be further described.

  First, when writing a file to the storage device 11, the file system view providing unit 14 on the master system 10 side writes block data by eliminating redundant storage as described above. At this time, it is determined whether or not the block data is deduplicated, and a deduplication flag indicating whether or not deduplication is performed is set in association with the address of the block data. Then, as shown in FIG. 5, metadata in which a deduplication flag is set for each block data address constituting the file is created and stored in the storage device 11. By examining this metadata, the master system 10 can recognize the block data stored in the storage device 11 without being deduplicated. As will be described later, the master system 10 classifies the B class or the D class. The size of block data corresponding to can be grasped.

  The master-side replication execution unit 12 (replication execution unit) communicates with the replica-side replication execution unit 22 (replication execution unit) via the network, and the data stored in the storage device 11 of the master system 10 is transferred to the replica system. A process of replicating to 20 storage devices 21 is performed. When the data stored in the storage device 11 is replicated, the master side replication execution unit 12 cooperates with the replica side replication execution unit 22 so that the data to be replicated is deduplicated in the master system 10. It is investigated whether or not it is stored and whether or not it is deduplicated and stored in the replica system 20 after replication. In particular, it is checked which of the following four types of data to be replicated corresponds to. The following classification table is shown in FIG.

A: At the time of writing in the master system, duplication is eliminated because it already exists in the master system, data is not actually written, and the same data also exists in the replica system. (Third classification)
B: At the time of writing in the master system, data was actually written because it did not exist in the master system, but the same data exists in the replica system. (First classification)
C: At the time of writing in the master system, duplication is eliminated because it already exists in the master system, and data is not actually written, but the same data does not exist in the replica system. (Second classification)
D: At the time of writing in the master system, data was actually written because it did not exist in the master system, and the same data does not exist in the replica system. (Fourth classification)

  Specifically, the master side replication execution unit 12 reads the metadata of the block data to be replicated from the storage device 11, and checks whether or not the block data is deduplicated from the deduplication flag. Further, the master side replication execution unit 12 reads out a hash value, which is a digest of the block data to be replicated, from the storage device 11 and makes a request for existence confirmation as to whether or not it is already stored in the replica side replication execution unit 22. In response to this, the replica-side replication execution unit 22 confirms whether or not block data having the same contents is stored in the storage device 21 from the hash value of the block data requested to be present, and the confirmation result is sent to the master side. Notify the replication execution unit 12.

  Then, the master-side replication execution unit 12 confirms the result of the above-described investigation, that is, the confirmation result received from the replica-side replication execution unit 22 (whether each block data corresponds to the A or B classification, or the C or D classification Each block data corresponds to any of A, B, C, and D based on the information that can be determined whether or not) and a deduplication flag corresponding to each block data in the metadata. Determine whether. At this time, the master-side replication execution unit 12 acquires the size (capacity) of the block data corresponding to each classification, adds the size of the block data corresponding to each classification for each classification, and the data classification information management unit 13 To store. Then, the master side replication execution unit 12 reads out the block data classified as C or D from the storage device 11 and transmits it to the replica side replication execution unit 22 for replication.

  Here, the data classification information management unit 13 holds the above-described counters for each of the classifications A, B, C, and D as classification information. This classification information is initialized at the start of replication.

  The progress information browsing unit 15 (progress rate calculation unit) receives a request for browsing the progress rate from the user system 30, calculates the progress rate, and provides the user system 30 with the user so that the user can browse. The progress rate is calculated by transferring the data amount (C ′ + D ′) transferred from the master system 10 to the replica system 20 from the start of replication to the time when the progress rate is calculated. It is calculated by dividing by the total amount of data (C + D) that can be newly stored without being deduplicated in the replica system 20. At this time, the data amount (B + D) of the data stored in the master system 10 classified as B or D without being deduplicated is determined as the data amount (B + C) determined at the time of progress rate calculation (B + D). By correcting with B ′, C ′), the total amount of data (C + D) that can be transferred from the master system 10 to the replica system 20 in the entire replication is estimated. A specific method of calculating the progress rate will be described in detail when the operation is described later.

[Operation]
Next, the operation of the above-described replication system will be described with reference to the flowcharts of FIGS. First, with reference to FIG. 7, an operation when a file is written in the master system 10 in response to a request from the user system 30 will be described.

  Upon receiving a file write request from the user system 30, the file system view providing unit 14 requests the storage device 11 to write (step S1). As described above, the writing process determines whether or not the block data having the same contents as the block data constituting the file related to the write request is already stored (step S2), and the already stored block data. Is stored in the storage device 11 if it does not exist (step S3).

  When the writing of the file is completed, a list of addresses of block data constituting the written file is received from the storage device 11, and deduplication indicating whether each is deduplicated (corresponding to A or C or B or D) Set flags and create file metadata. Then, the file metadata is written in the storage device 11 (step S4).

  Next, an operation at the time of executing replication will be described with reference to FIG. The master-side replication execution unit 12 requests the data classification information management unit 13 to start recording classification information when replication is started. In response to the request to start recording, the data classification information management unit 13 prepares initialized classification information (a counter group cleared to zero) (step S11).

  Subsequently, the master-side replication execution unit 12 requests the storage device 11 to read the digest (hash value) of the block data to be replicated, and acquires the digest (step S12). Then, the master-side replication execution unit 12 sends a digest of the block data to be replicated to the replica-side replication execution unit 22, and requests the presence of the block data to be replicated (step S13).

  The replica-side replication execution unit 22 passes the block data digest received from the master-side replication execution unit 12 to the storage device 21 and confirms whether the same block data is possessed. Then, the replica side replication execution unit 22 notifies the master side replication execution unit 12 of the existence confirmation result.

  The master side replication execution unit 12 confirms whether or not the block data exists in the replica system 20 from the replica side replication execution unit 22, that is, whether each block data hits A or B, C or D Receive information that can be determined. Further, the master side replication execution unit 12 acquires the metadata from the storage device 11, and based on the deduplication flag corresponding to each block data in the metadata, whether the block data is deduplicated in the master system. It is confirmed whether or not (step S14). Then, the master side replication execution unit 12 determines that each block data is A, B, C, based on the existence confirmation result of the replica side replication execution unit 22 and the confirmation result of deduplication by the master side replication execution unit 12. Which of D is classified is determined (step S15).

  Subsequently, the master-side replication execution unit 12 requests the data classification information management unit 13 to add the size of the block data to the counter corresponding to the determined block data classification (step S16). Then, block data classified as C or D, that is, block data that does not exist in the replica system 20 is read from the storage device 11 by the master side replication execution unit 12 and transmitted to the replica side replication execution unit 22. .

  The replica-side replication execution unit 22 requests the storage device 21 to write the block data received from the master-side replication execution unit 12, and writes to the storage device 21. Then, the processing from step S12 to step S16 described above is performed for all block data to be replicated (step S17).

  Next, the operation at the time of providing the progress rate will be described with reference to FIG. The progress information browsing unit 15 receives a request for acquisition of the progress rate from the user system 30 during the execution of replication, and requests the master side replication execution unit 12 to acquire the progress rate (step S21). The master-side replication execution unit 12 requests the data classification information management unit 13 to acquire classification information, acquires the counter value at that time for each of A, B, C, and D, and progresses according to the following formula: The rate is calculated and returned to the progress information browsing unit 15. The following progress rate calculation may be executed by the progress information browsing unit 15.

Progress rate = (C ′ + D ′) ÷ ((B + D) − ((B ′ + D ′) ÷ (B + D)) × (B ′ ÷ (B ′ + D ′)) × (B + D) + ((A ′ + C ′ ) ÷ (A + C)) × (C ′ ÷ (A ′ + C ′)) × (A + C))
= (C ′ + D ′) ÷ (B + D−B ′ + C ′)

  In the above formula, A, B, C, and D mean the final data amount of each corresponding classification (the total result at the time of completion of replication). A ′, B ′, C ′, and D ′ indicate the classification information of each corresponding classification acquired from the data classification information management unit 13 at a certain point in time during execution of replication (for example, when the progress rate is requested). means.

Note that some of the above expressions represent the following contents.
(B ′ + D ′) ÷ (B + D): Progress rate of processing of block data not deduplicated at the time of writing in the master system 10 (A ′ + C ′) ÷ (A + C): Deduplicated at the time of writing in the master system 10 Progress rate of block data processing B ′ ÷ (B ′ + D ′): Ratio of block data that has undergone replication processing and has been deduplicated at replica site 20 among block data that has not been deduplicated at the time of writing by master system 10 C ′ ÷ (A ′ + C ′)) × (A + C): Ratio of block data that has been subjected to replication processing and not deduplicated at the replica site out of block data that has been deduplicated at the time of writing in the master system

  In the above formula, most of the data classified into A and D exists, and B and C hardly exist. That is, most of the data stored in the replica system 20 is a copy of the data stored in the master system 10. Under the assumption that there is, first, the predicted value of the processing amount at the start of replication is assumed to be “B + D”. That is, since B and C hardly exist, it is considered that the processing amount C + D≈B + D in the entire replication. The value “B + D” can be obtained by referring to the deduplication flag in the master system 10. Then, based on information (A ′, B ′, C ′, D ′) obtained in the course of processing, the predicted value “B + D” is corrected, and the transfer data amount (C ′ + D ′) at that time is corrected. The progress rate is calculated by dividing.

  Specifically, the processing amount prediction value “B + D” is corrected as follows. First, for data classified into B or D from the initial value “B + D”, the ratio (B ′ ÷ (B ′ + D ′)) of data that has been deduplicated is multiplied by the total amount (B + D), and B and D In order to reduce the influence of fluctuation due to the occurrence bias, the number obtained by multiplying the processing completion ratio ((B ′ + D ′) ÷ (B + D)) of data belonging to B and D is subtracted. Further, for data classified as A or C with respect to the initial value “B + D”, the ratio (A ′ ÷ (A ′ + C ′)) of deduplicated data is multiplied by the total amount (A + C), and A and In order to reduce the influence of fluctuation due to the bias in the occurrence of C, a number obtained by multiplying the processing completion ratio ((A ′ + C ′) ÷ (A + C)) of data belonging to A and C is added.

  The method for correcting the predicted value of the processing amount (C + D) in the above calculation is equivalent to subtracting “B ′” obtained in the replication process from the initial value “B + D” and adding “C ′”. Therefore, “C + D” that is a true processing amount can be appropriately expressed.

  As described above, according to the present invention, in the process of replication processing, the processed data is classified into the above A, B, C, and D, and totalized, and the processing amount “C + D” is based on this data. The estimated value of is obtained and the progress rate is calculated. Therefore, the progress rate closer to the actual situation can be obtained.

<Appendix>
Part or all of the above-described embodiment can be described as in the following supplementary notes. The outline of the configuration of the replication system (see FIG. 10), program, and replication method in the present invention will be described below. However, the present invention is not limited to the following configuration.

(Appendix 1)
The master-side storage system 100 and the replica-side storage system 200 each having the storage devices 110 and 210 store the data in the storage device, and other data having the same data content as the data already stored in the storage device Is stored in the storage device, the data already stored in the storage device is referred to as other data, the function to eliminate the duplicate storage,
Replication execution units 101 and 201 for replicating data stored in the storage device of the master storage system to the storage device of the replica storage system;
A progress rate calculation unit 102 that calculates the progress rate of the replication at a predetermined replication time point,
The replication execution units 101 and 201 investigate whether or not the data to be replicated from the master storage system to the replica storage system is deduplicated and stored in the storage device of the replica storage system. Transfer the data stored in the replica side storage system without being deduplicated among the data to be replicated from the master side storage system to the replica side storage system,
The progress rate calculation unit 102 is transferred from the master side storage system to the replica side storage system in the entire replication based on the investigation result of the data to be replicated, and is not deduplicated to the storage device of the replica side storage system. Based on the estimated total data amount and the amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point. To calculate the replication progress rate at the given replication point,
Replication system.

(Appendix 2)
The replication system according to attachment 1, wherein
The replication execution unit determines whether data to be replicated from the master storage system to the replica storage system is deduplicated and stored in a storage device of the master storage system, and the replica storage system Whether or not it is deduplicated and stored in the storage device and whether it belongs to any of the classifications set according to, and calculates the data amount of the data belonging to each of the classifications,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the data amount for each classification to which the data to be replicated belongs, and is duplicated in the storage device of the replica side storage system. Estimate the total amount of data newly stored without being excluded,
Replication system.

(Appendix 3)
The replication system according to appendix 2,
The replication execution unit stores data to be replicated from the master storage system to the replica storage system without being deduplicated in the storage device of the master storage system. The first classification (B) that is deduplicated and stored in the storage device, and the deduplication is stored in the storage device of the master storage system, but the storage device of the replica storage system And at least investigating whether it belongs to the second classification (C) that is stored without being deduplicated, and calculating the amount of data belonging to each of the first classification and the second classification,
Replication system.

(Appendix 4)
The replication system according to appendix 2,
The replication execution unit stores the data to be replicated from the master storage system to the replica storage system in the storage device of the master storage system without being deduplicated, but the storage device of the replica storage system Is deduplicated and stored in the storage device of the master-side storage system, but is deduplicated and stored in the storage device of the replica-side storage system. Is stored without being deduplicated and stored in the storage device of the master-side storage system, and deduplicated and stored in the storage device of the replica-side storage system. The third classification (A), and the master storage system And whether it belongs to the fourth classification (D) that is stored in the storage device of the replica-side storage system and not stored in the storage device of the replica-side storage system. Calculating the amount of data belonging to each of the classifications;
Replication system.

(Appendix 5)
The replication system according to appendix 3 or 4,
The replication execution unit calculates the amount of data to be replicated investigated at a predetermined replication point for each classification,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the whole replication based on the data amount of the first classification and the second classification examined at a predetermined replication time point, and the replica Estimate the total amount of data that is newly stored in the storage device of the storage system without being deduplicated,
Replication system.

(Appendix 6)
The replication system according to appendix 5,
The progress rate calculation unit subtracts the data amount of the first classification from the data amount of data stored in the storage device of the master side storage system without being deduplicated, and the data of the second classification Estimated as the total amount of data transferred from the master storage system to the replica storage system and newly stored in the storage device of the replica storage system without deduplication The progress rate is calculated by dividing the transfer amount of the data transferred from the master storage system to the replica storage system at a predetermined replication time by the estimated total data amount.
Replication system.

(Appendix 7)
The replication system according to any one of appendices 1 to 6,
The replication execution unit performs an investigation of data to be replicated based on data feature amount information representing a feature amount of the data based on the content of the data,
Replication system.

(Appendix 8)
The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In the case of providing a function for eliminating duplicate storage by referring to data already stored in the storage device as other data, between the master-side storage system and the replica-side storage system. In the information processing device that controls replication in
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication time point, and
The replication execution unit investigates whether or not data to be replicated from the master side storage system to the replica side storage system is deduplicated and stored in the storage device of the replica side storage system, and performs replication. Transfer data stored in the replica storage system without being deduplicated from the master storage system to the replica storage system,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the investigation result of the data to be replicated and is not deduplicated in the storage device of the replica side storage system. Estimating the total data amount of newly stored data, and based on the estimated total data amount and the data amount of data transferred from the master storage system to the replica storage system at a predetermined replication point , Calculate the replication progress rate at the given replication point,
A program to make things happen.

(Appendix 8-2)
The program according to attachment 8, wherein
The replication execution unit determines whether data to be replicated from the master storage system to the replica storage system is deduplicated and stored in a storage device of the master storage system, and the replica storage system Whether or not it is deduplicated and stored in the storage device and whether it belongs to any of the classifications set according to, and calculates the data amount of the data belonging to each of the classifications,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the data amount for each classification to which the data to be replicated belongs, and is duplicated in the storage device of the replica side storage system. Estimate the total amount of data newly stored without being excluded,
program.

(Appendix 9)
The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device Replication function between the master side storage system and the replica side storage system, which has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data. A method,
A replication execution step of replicating the data stored in the storage device of the master side storage system to the storage device of the replica side storage system;
A progress rate calculating step for calculating the progress rate of the replication at a predetermined replication point;
The replication execution step investigates whether data to be replicated from the master side storage system to the replica side storage system is stored in the storage device of the replica side storage system in a deduplicated manner and performs the replication. Transfer data stored in the replica storage system without being deduplicated from the master storage system to the replica storage system,
The progress rate calculation step is based on the investigation result of the data to be replicated, and is transferred from the master side storage system to the replica side storage system in the entire replication and is not deduplicated in the storage device of the replica side storage system. Estimating the total data amount of newly stored data, and based on the estimated total data amount and the data amount of data transferred from the master storage system to the replica storage system at a predetermined replication point , Calculate the replication progress rate at the given replication point,
Replication method.

(Appendix 10)
The replication method according to appendix 9, wherein
In the replication execution step, whether the data to be replicated from the master side storage system to the replica side storage system is deduplicated and stored in the storage device of the master side storage system, and the replica side storage system Whether or not it is deduplicated and stored in the storage device and whether it belongs to any of the classifications set according to, and calculates the data amount of the data belonging to each of the classifications,
The progress rate calculating step is based on the data amount for each classification to which the data to be replicated belongs, and is transferred from the master side storage system to the replica side storage system in the entire replication and overlapped with the storage device of the replica side storage system. Estimate the total amount of data newly stored without being excluded,
Replication method.

  Note that the above-described program is stored in a storage device or recorded on a computer-readable recording medium. For example, the recording medium is a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

  Although the present invention has been described with reference to the above-described embodiment and the like, the present invention is not limited to the above-described embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Storage system 2 Accelerator node 3 Storage node 4 Backup system 10 Master system 11 Storage device 12 Master side replication execution part 13 Data classification information management part 14 File system view provision part 15 Progress information browsing part 20 Replica system 21 Storage apparatus 22 Replica side Replication execution unit 100 Master side storage systems 101 and 201 Replication execution unit 102 Progress rate calculation units 110 and 210 Storage device 200 Replica side storage system

Claims (7)

The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In this case, it has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data,
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication point;
The replication execution unit stores the data to be replicated from the master storage system to the replica storage system in the storage device of the master storage system without being deduplicated, but the storage device of the replica storage system Is stored in the storage device of the master-side storage system, but is not deduplicated in the storage device of the replica-side storage system. A second classification that is stored, and a third classification that is deduplicated and stored in the storage device of the master-side storage system and is deduplicated and stored in the storage device of the replica-side storage system And duplicated in the storage device of the master storage system And whether the data belongs to the fourth classification that is stored without being deduplicated and stored in the storage device of the replica-side storage system, and the amount of data belonging to each classification Further, the data stored without being deduplicated in the replica-side storage system among the data to be replicated is transferred from the master-side storage system to the replica-side storage system,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the data amount for each classification to which the data to be replicated belongs, and is duplicated in the storage device of the replica side storage system. Estimating the total amount of data newly stored without being excluded, the estimated total amount of data, and the amount of data transferred from the master storage system to the replica storage system at a predetermined replication point Based on, the replication progress rate at the predetermined replication point is calculated ,
further,
The replication execution unit calculates the amount of data to be replicated investigated at a predetermined replication point for each classification,
The progress rate calculation unit is configured to store the data stored in the storage device of the master storage system without deduplication based on the data amounts of the first classification and the second classification examined at a predetermined replication time point. A value obtained by subtracting the data amount of the first classification and adding the data amount of the second classification from the data amount is transferred from the master side storage system to the replica side storage system in the entire replication, and the replica Estimated as the total amount of data that is newly stored in the storage device of the side storage system without being deduplicated, and the transfer amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point By dividing by the estimated total data amount Calculating the progress rate,
Replication system. The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In this case, it has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data,
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication point;
The replication execution unit stores data to be replicated from the master storage system to the replica storage system without being deduplicated in the storage device of the master storage system. The first classification, which is deduplicated and stored in the storage device, is deduplicated and stored in the storage device of the master storage system, but is deduplicated in the storage device of the replica storage system The at least one of the second classifications stored in the database, and at least investigating whether the data belongs to each of the first classification and the second classification, and calculating the amount of data belonging to each of the first classification and the second classification. The amount of data to be Calculated, further, it transfers the data stored without being deduplication on the replica side storage system among the data to be the replication from the master side storage system to the replica side storage system,
The progress rate calculation unit is configured to store the data stored in the storage device of the master storage system without deduplication based on the data amounts of the first classification and the second classification examined at a predetermined replication time point. A value obtained by subtracting the data amount of the first classification and adding the data amount of the second classification from the data amount is transferred from the master side storage system to the replica side storage system in the entire replication, and the replica Estimated as the total amount of data that is newly stored in the storage device of the side storage system without being deduplicated, and the transfer amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point By dividing by the estimated total data amount Calculating the progress rate,
Replication system. The replication system according to claim 1 or 2 ,
The replication execution unit performs an investigation of data to be replicated based on data feature amount information representing a feature amount of the data based on the content of the data,
Replication system. The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In the case of providing a function for eliminating duplicate storage by referring to data already stored in the storage device as other data, between the master-side storage system and the replica-side storage system. In the information processing device that controls replication in
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication time point, and
The replication execution unit stores the data to be replicated from the master storage system to the replica storage system in the storage device of the master storage system without being deduplicated, but the storage device of the replica storage system Is stored in the storage device of the master-side storage system, but is not deduplicated in the storage device of the replica-side storage system. A second classification that is stored, and a third classification that is deduplicated and stored in the storage device of the master-side storage system and is deduplicated and stored in the storage device of the replica-side storage system And duplicated in the storage device of the master storage system And whether the data belongs to the fourth classification that is stored without being deduplicated and stored in the storage device of the replica-side storage system, and the amount of data belonging to each classification Further, the data stored without being deduplicated in the replica-side storage system among the data to be replicated is transferred from the master-side storage system to the replica-side storage system,
The progress rate calculation unit is transferred from the master side storage system to the replica side storage system in the entire replication based on the data amount for each classification to which the data to be replicated belongs, and is duplicated in the storage device of the replica side storage system. Estimating the total amount of data newly stored without being excluded, the estimated total amount of data, and the amount of data transferred from the master storage system to the replica storage system at a predetermined replication point Based on, the replication progress rate at the predetermined replication point is calculated ,
further,
The replication execution unit calculates the amount of data to be replicated investigated at a predetermined replication point for each classification,
The progress rate calculation unit is configured to store the data stored in the storage device of the master storage system without deduplication based on the data amounts of the first classification and the second classification examined at a predetermined replication time point. A value obtained by subtracting the data amount of the first classification and adding the data amount of the second classification from the data amount is transferred from the master side storage system to the replica side storage system in the entire replication, and the replica Estimated as the total amount of data that is newly stored in the storage device of the side storage system without being deduplicated, and the transfer amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point By dividing by the estimated total data amount Calculating the progress rate,
A program to make things happen. The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device In the case of providing a function for eliminating duplicate storage by referring to data already stored in the storage device as other data, between the master-side storage system and the replica-side storage system. In the information processing device that controls replication in
A replication execution unit that replicates data stored in the storage device of the master-side storage system to the storage device of the replica-side storage system;
A progress rate calculation unit that calculates the progress rate of the replication at a predetermined replication time point, and
The replication execution unit stores data to be replicated from the master storage system to the replica storage system without being deduplicated in the storage device of the master storage system. The first classification, which is deduplicated and stored in the storage device, is deduplicated and stored in the storage device of the master storage system, but is deduplicated in the storage device of the replica storage system The at least one of the second classifications stored in the database, and at least investigating whether the data belongs to each of the first classification and the second classification, and calculating the amount of data belonging to each of the first classification and the second classification. The amount of data to be Calculated, further, it transfers the data stored without being deduplication on the replica side storage system among the data to be the replication from the master side storage system to the replica side storage system,
The progress rate calculation unit is configured to store the data stored in the storage device of the master storage system without deduplication based on the data amounts of the first classification and the second classification examined at a predetermined replication time point. A value obtained by subtracting the data amount of the first classification and adding the data amount of the second classification from the data amount is transferred from the master side storage system to the replica side storage system in the entire replication, and the replica Estimated as the total amount of data that is newly stored in the storage device of the side storage system without being deduplicated, and the transfer amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point By dividing by the estimated total data amount Calculating the progress rate,
A program to make things happen. The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device Replication function between the master side storage system and the replica side storage system, which has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data. A method,
A replication execution step of replicating the data stored in the storage device of the master side storage system to the storage device of the replica side storage system;
A progress rate calculating step for calculating the progress rate of the replication at a predetermined replication point;
In the replication execution step, data to be replicated from the master storage system to the replica storage system is stored in the storage device of the master storage system without being deduplicated, but the storage device of the replica storage system Is stored in the storage device of the master-side storage system, but is not deduplicated in the storage device of the replica-side storage system. A second classification that is stored, and a third classification that is deduplicated and stored in the storage device of the master-side storage system and is deduplicated and stored in the storage device of the replica-side storage system To the storage device of the master storage system. And whether the data belongs to the fourth category that is stored without being excluded and stored in the storage device of the replica-side storage system without being deduplicated, and the data amount of the data belonging to each category. Further, the data stored without being deduplicated in the replica-side storage system among the data to be replicated is transferred from the master-side storage system to the replica-side storage system,
The progress rate calculating step is based on the data amount for each classification to which the data to be replicated belongs, and is transferred from the master side storage system to the replica side storage system in the entire replication and overlapped with the storage device of the replica side storage system. Estimating the total amount of data newly stored without being excluded, the estimated total amount of data, and the amount of data transferred from the master storage system to the replica storage system at a predetermined replication point Based on, the replication progress rate at the predetermined replication point is calculated ,
further,
The replication execution step calculates the amount of data to be replicated investigated at a predetermined replication point for each classification,
In the progress rate calculation step, the data stored in the storage device of the master storage system without being deduplicated based on the data amounts of the first classification and the second classification examined at a predetermined replication time point. A value obtained by subtracting the data amount of the first classification and adding the data amount of the second classification from the data amount is transferred from the master side storage system to the replica side storage system in the entire replication, and the replica Estimated as the total amount of data that is newly stored in the storage device of the side storage system without being deduplicated, and the transfer amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point Divide by the estimated total amount of data , To calculate the progress rate,
Replication method. The master-side storage system and replica-side storage system each having a storage device store the data in the storage device, and store other data in the storage device with the same data content as the data already stored in the storage device Replication function between the master side storage system and the replica side storage system, which has a function of eliminating duplicate storage by referring to data already stored in the storage device as other data. A method,
A replication execution step of replicating the data stored in the storage device of the master side storage system to the storage device of the replica side storage system;
A progress rate calculating step for calculating the progress rate of the replication at a predetermined replication point;
In the replication execution step, data to be replicated from the master storage system to the replica storage system is stored without being deduplicated in the storage device of the master storage system. The first classification, which is deduplicated and stored in the storage device, is deduplicated and stored in the storage device of the master storage system, but is deduplicated in the storage device of the replica storage system The at least one of the second classifications stored in the database, and at least investigating whether the data belongs to each of the first classification and the second classification, and calculating the amount of data belonging to each of the first classification and the second classification. The amount of data to be Calculated, further, it transfers the data stored in the data that the replication without being deduplication on the replica side storage system from the master side storage system to the replica side storage system,
In the progress rate calculation step, the data stored in the storage device of the master storage system without being deduplicated based on the data amounts of the first classification and the second classification examined at a predetermined replication time point. A value obtained by subtracting the data amount of the first classification and adding the data amount of the second classification from the data amount is transferred from the master side storage system to the replica side storage system in the entire replication, and the replica Estimated as the total amount of data that is newly stored in the storage device of the side storage system without being deduplicated, and the transfer amount of data transferred from the master side storage system to the replica side storage system at a predetermined replication point Divide by the estimated total amount of data , To calculate the progress rate,
Replication method.

Images