JP6113816B1 - Information processing system, information processing apparatus, and program - Google Patents

Abstract

Data deduplication is performed in an information processing system. A data storage destination device includes: first storage data among a plurality of storage data having a storage unit size generated based on the data; and a first fragment having an overlap detection size larger than the storage unit size. A first hash value corresponding to the data and a second hash value corresponding to at least part of the first stored data are received. Second storage data having a storage unit size generated based on the storage target data by the external device 2 is received. Corresponds to the first fragment data when the second hash value corresponding to at least part of the first stored data matches the third hash value corresponding to at least part of the second stored data. When the first hash value matches the fourth hash value stored in a plurality of devices and corresponding to the second fragment data including the second stored data and corresponding to the second fragment data having the duplicate detection size, The duplication with respect to 2nd fragment data is detected. [Selection] Figure 1

Description

  The present embodiment relates to an information processing system, an information processing apparatus, and a program.

  In an object storage used for large-scale storage, an erasure code is used to realize fault tolerance.

US Pat. No. 7,992,037 JP 2010-79886 A

Rabin, Michael O. Fingerprinting by random polynomials.Center for Research in Computing Techn., Aiken Computation Laboratory, Univ., 1981. openstack CLOUD SOFTWARE, “The Rings − swift 2.5.1.dev128 documentation-Open Stack Docs”, [searched November 18, 2015], Internet <URL: http://docs.openstack.org/developer/swift/overview_ring .html>

  The present embodiment provides an information processing system, an information processing apparatus, and a program for performing deduplication of data.

  The information processing system according to the present embodiment includes a first information processing device and a plurality of second information processing devices. The first information processing apparatus includes a generation unit, a calculation unit, and a transmission unit. The generation unit generates a plurality of storage data having a storage unit size based on the data. The calculation unit calculates a first hash value corresponding to the first fragment data included in the data and having a duplicate detection size larger than the storage unit size, and the first hash data included in the first fragment data. A second hash value corresponding to at least a part of the stored data is calculated. The transmission unit transmits the first storage data, the first hash value, and the second hash value to a storage destination device among the plurality of second information processing devices. The storage destination device includes a receiving unit and a processing unit. The receiving unit receives the first storage data, the first hash value, and the second hash value, and stores the second storage of the storage unit size generated based on the storage target data by the external information processing apparatus. Receive data. Whether the processing unit matches the second hash value corresponding to at least part of the first stored data with the third hash value corresponding to at least part of the second stored data received by the receiving unit If the second hash value matches the third hash value, the first hash value corresponding to the first fragment data is stored in the plurality of second information processing devices. It is determined whether the fourth hash value corresponding to the second fragment data of the duplicate detection size including the second stored data matches, and if the first hash value matches the fourth hash value, the second Detect duplicates of fragment data.

1 is a block diagram illustrating the configuration of an information processing system according to a first embodiment. 1 is a conceptual diagram illustrating the configuration of an information processing system according to a first embodiment. The conceptual diagram which shows an example of the relationship between an object and the disk of a fragment server. The figure which illustrates a disk selection index file. The figure which illustrates the 1st thru | or 3rd system of the de-duplication method. The figure which illustrates the comparison result of normal hash calculation and rolling hash calculation. The flowchart which illustrates a fragment preservation | save process. The conceptual diagram which illustrates a fragment preservation | save process. The flowchart which illustrates the read-out procedure of an object. The figure which shows the specific example of the removal process of the overlapping fragment. The figure which illustrates a fingerprint set. The figure which illustrates the preservation | save format and search method of a fingerprint set. The flowchart which illustrates the duplication removal process in a free position. The figure which illustrates the 1st reading process of the fragment in a deduplication process. The figure which illustrates the 2nd reading process of the fragment in a deduplication process. The figure which shows the 1st example of the duplication removal process in a free position. The figure which shows the 2nd example of the duplication removal process in a free position. The figure which shows the 1st example of the rearrangement of the object by a front end server. The figure which shows the 2nd example of the rearrangement of the object by a front end server.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, substantially the same or substantially the same functions and components are denoted by the same reference numerals and will be described as necessary.

[First Embodiment]
The object storage used in large-scale storage manages the data to be saved in an abstracted object unit. For example, the type and number of hardware at the save destination such as HDD (Hard Disk Drive) or SSD (Solid State Drive) Is not limited by the file system specifications or file system specifications. Therefore, the object storage has high expandability such that hardware replacement and capacity expansion can be easily performed, and a large-scale storage system can be constructed at low cost.

  The information processing system according to the present embodiment uses object storage and achieves high fault tolerance using erasure codes. In addition, the information processing system according to the present embodiment applies deduplication of data stored in the entire system while maintaining the reliability of the erasure code, thereby reducing the storage capacity, in other words, improving the storage efficiency.

  In this embodiment, reliability against a physical disk failure is maintained by applying deduplication while managing a set of objects and erasure codes distributed and stored in a plurality of physical disks.

  The information processing system according to the present embodiment assumes the use of a highly functional drive such as an IP (Internet Protocol) connection type drive in order to maintain the expandability of the storage capacity. However, it may be a drive that can be connected to a plurality of devices other than the IP connection. The drive may include, for example, a nonvolatile semiconductor memory. The nonvolatile memory is, for example, a NAND flash memory, but a NOR flash memory, an MRAM (Magnetoresistive Random Access Memory), a PRAM (Phase change Random Access Memory), a ReRAM (Resistive Random Access Memory). : Resistance change type memory), FeRAM (Ferroelectric Random Access Memory), and other nonvolatile semiconductor memories. For example, the non-volatile memory may be another non-volatile memory or magnetic memory that is not a non-volatile semiconductor memory. For example, the nonvolatile memory may be a flash memory having a three-dimensional structure. For example, the non-volatile memory may be a disk (for example, a disc or a disk).

  In the information processing system according to the present embodiment, a plurality of drives share the search for overlapping positions. This reduces the load on the front-end server with respect to an increase in the data storage amount.

  In this embodiment, a terminal used by a user (or an application running on the terminal) stores or acquires an object using a predetermined rule such as REST (Representational State Transfer) API (Application Programming Interface). To do. The information processing apparatus in charge of this API is called a front end server. When the front-end server receives an object storage request from the user's terminal or the application of the terminal used by the user, the front-end server decomposes the object into an appropriate number of fragments (hereinafter referred to as a stride), and erasure codes for each fragment. Encode using. The stride has a duplicate detection size.

  The stride is divided into minimum division units (hereinafter referred to as fragments) to which erasure codes are applied by encoding. A fragment has a storage unit size. The front-end server stores these fragments in a plurality of fragment servers that are final data storage destinations using, for example, a KVS (Key-Value Store) API.

  Thereafter, the fragment server performs deduplication on the stored fragments. For example, deduplication is performed in units of strides in order to maintain the reliability of erasure codes.

  Note that the API used when transmitting / receiving data between the terminal used by the user and the front-end server and the API used when transmitting / receiving data between the front-end server and the fragment server are not limited to those described above. .

  FIG. 1 is a block diagram illustrating the configuration of the information processing system according to this embodiment.

  The information processing system 1 includes one or more front-end servers 2 and a plurality of fragment servers 3. In FIG. 1, a case where there is one front-end server 2 will be described. Further, one fragment server among the plurality of fragment servers 3 will be described as a representative.

  The front-end server 2 generates a plurality of strides from the object, generates a plurality of fragments from each of the plurality of strides, and distributes the plurality of fragments to the plurality of fragment servers 3 for transmission.

  Each of the plurality of fragment servers 3 determines whether or not at least a part of the fragments stored in the storage unit 31 of the own machine matches at least a part of the received fragments. Each of the plurality of fragment servers 3 stores the stored fragment in the plurality of fragment servers 3 when at least a part of the stored fragment matches at least a part of the received fragment. It is determined whether or not the data for one stride that is received matches the data for one stride that includes the received fragment and is included in the object. Then, each of the plurality of fragment servers 3 sets the overlapping position information indicating the overlapping position in the object when the stored data for one stride matches the data for one stride included in the object. Transmit to the front-end server 2.

  The front end server 2 regenerates a plurality of strides from the object based on the overlapping position information so that the overlapping portion is included in the same stride, and further generates a plurality of fragments from each of the regenerated plurality of strides. The regenerated fragments are transmitted to the plurality of fragment servers 3 in a distributed manner.

  Each of the plurality of fragment servers 3 performs deduplication when the regenerated and received fragment matches the fragment stored in its own storage unit 31.

  The front end server 2 includes a controller 20 and a storage unit 21.

  The controller 20 includes a transmission / reception unit 22, a processor 23, a memory 24, and a control unit 25.

  The transmission / reception unit 22 transmits / receives, for example, commands, addresses, data, information, instructions, signals, and the like to / from the terminal 11 (or application running on the terminal 11) used by the user U and the fragment server 3.

  The terminal 11 used by the user U may be, for example, a computer, a PDA (Personal Digital Assistant), a smartphone, a tablet terminal, or the like.

  The processor 23 executes control processing and arithmetic processing based on instructions from the transmission / reception unit 22 and the control unit 25. As the processor 1, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP) is used.

  The memory 24 is a main storage device and follows control from the processor 23. The memory 24 temporarily stores data transmitted / received by the transmitting / receiving unit 22 or data generated by the control unit 25 based on the control of the processor 23.

  The control unit 25 includes an object dividing unit 201, an erasure code calculating unit 202, and a fingerprint calculating unit 203.

  The object dividing unit 201 divides an object received from the terminal 11 used by the user U through the transmission / reception unit 22 into strides. The size of the stride is arbitrarily determined by the system.

  If the object size is not an integer multiple of the stride size, padding data (for example, zero data) that has no meaning is added to the object until it becomes an integer multiple, that is, padding is performed. Note that padding may be realized by dividing an object into a plurality of strides and supplementing padding data to strides that are less than a predetermined size among the divided strides.

  The erasure code calculation unit 202 performs encoding that calculates an erasure code for each stride obtained by the object division unit 201. Specifically, the erasure code calculation unit 202 generates a plurality of fragments based on the stride. The plurality of fragments corresponding to the stride include an information symbol part and a parity symbol part. Details of the erasure code will be described later with reference to FIG.

  The fingerprint calculation unit 203 calculates a fingerprint for arbitrary data. The calculation target data may be, for example, a stride unit, a fragment unit, a half unit of a fragment (referred to as a 1/2 fragment unit), or the like. The fingerprint is, for example, a hash value, and is a value that is uniquely determined based on the content of data. That is, for example, the stride fingerprints made of the same data are always the same. Therefore, if the fingerprints are the same, there is a high possibility that they are the same data. Therefore, fingerprint comparison is effective when searching for duplicate data in the deduplication process. In this embodiment, by comparing fingerprints corresponding to a plurality of data, it is determined whether or not the plurality of data match.

  In addition, the fingerprint calculation unit 203 stores the calculated fingerprint in the storage unit 21.

  The storage unit 21 includes an object storage unit 211, a fingerprint set 212, and a disk selection index file 213.

  The object storage unit 211 stores the object received by the transmission / reception unit 22.

  The fingerprint set 212 is a collection of fingerprints for each stride or fragment used in the deduplication process. The fingerprint set 212 may be a collection of one or more files, for example.

  The fingerprint set 212 is transmitted from the transmission / reception unit 22 of the controller 20 to the fragment server 3 when a certain amount of fingerprints are recorded.

  The disk selection index file 213 indicates a set of fragment servers that are fragment storage destinations. The disk selection index file 213 associates a reference hash value with storage destination identification information determined corresponding to the reference hash value. The control unit 25 determines the storage destination of each fragment by referring to the disk selection index file 213 using a value determined based on the fingerprint calculated by the fingerprint calculation unit 203 as a key. The calculation of the key information for referring to the disc selection index file 213 from the fingerprint may be performed using, for example, a hash function.

  The determination of the fragment storage destination using the disk selection index file 213 will be described later with reference to FIG. Note that the disk selection index file 213 may not be stored in a file format.

  The fragment server 3 includes a controller 30 and a storage unit 31.

  The controller 30 includes a transmission / reception unit 32, a processor 33, a memory 34, and a control unit 35.

  The transmission / reception unit 32 performs transmission / reception of commands, addresses, data, information, instructions, signals, and the like with the front-end server 2.

  The processor 33 executes control processing and arithmetic processing based on instructions from the transmission / reception unit 32 and the control unit 35. As the processor 33, for example, a CPU, MPU, DSP, or the like is used as in the processor 23.

  The memory 34 is a main storage device and follows control from the processor 33. The memory 34 temporarily stores data transmitted / received by the transmission / reception unit 32 or data generated by the control unit 35 based on the control of the processor 33.

  The control unit 35 includes a deduplication processing unit 301.

  The deduplication processing unit 301 uses the fingerprint set 311 to search for duplicate portions of the fragments stored in the fragment storage unit 312 and performs deduplication.

  In addition, since deduplication is performed in units of strides, the front-end server 2, for example, if the starting point of the overlapping part exists in the middle of the stride, in order to make the starting point of the overlapping part coincide with the starting point of the overlapping part, Rearrange the stride by dividing the stride at the starting point. As described above, when it becomes necessary to rearrange the strides for deduplication, the deduplication processing unit 301 transmits a stride rearrangement instruction to the front-end server 2. Details of the deduplication processing will be described later.

  The storage unit 31 includes a fingerprint set 311, a fragment storage unit 312, and overlapping position information 313. Hereinafter, in the present embodiment, the storage unit 31 is a disk. However, using the storage unit 31 as a disk does not mean that the storage unit 31 is limited to a disk. Various nonvolatile memories can be used as the storage unit 31.

  The fingerprint set 311 is received by the storage unit 31 from the front end server 2 via the transmission / reception unit 32 of the controller 30 of the fragment server 3 and stored in the storage unit 31. The fingerprint set 311 is referred to when deduplication is performed.

  The fragment storage unit 312 stores the fragment received from the front-end server 2 in the storage unit 31 via the transmission / reception unit 32 of the controller 30 of the fragment server 3.

  The duplicate position information 313 includes information indicating the existence of duplicate data in the object to be saved, for example, the position of duplicate data that is referenced when the front-end server 2 executes stride rearrangement when duplicate data exists. More specifically, the duplication position information 313 includes the origin of duplicate data that is determined to have duplicate data for one stride as a result of duplication detection performed by the duplicate elimination processing unit 301. When a certain amount of the overlapping position information 313 is stored, the control unit 35 transmits the information to the front end server 2 through the transmission / reception unit 32.

  Details of the deduplication processing will be described later.

  FIG. 2 is a conceptual diagram illustrating the configuration of the information processing system 1 according to this embodiment.

  The information processing system 1 includes a plurality of front end servers 2 and a plurality of fragment servers 3. By adding the front-end server 2, the processing capability of data received from the terminal 11 and the fragment server 3 used by the user U is improved, that is, the processing capability of the information processing system 1 as a whole is improved.

  The terminal 11 of the user U or the application of the terminal 11 sends the object OB to any one of the plurality of front end servers 2 based on, for example, the REST API. The object may be a file, for example.

  In the present embodiment, the fragment server 3 is configured with a number equal to or greater than the number of fragments generated from the stride after erasure coding. Further, by adding the fragment server 3, the final storage destination of the object OB increases, that is, the storage capacity of the information processing system 1 as a whole increases.

  First, the front end server 2 divides the object OB into a plurality of strides ST. Next, the front-end server 2 encodes each stride ST with an erasure code to generate a plurality of fragments. The plurality of fragments include an information symbol part IS and a parity symbol part PS. In FIG. 2, an information symbol part IS including four fragments and a parity symbol part PS including two fragments are generated from one stride. The number of fragments included in the information symbol part IS and the number of fragments included in the parity symbol part PS can be changed by one or more.

  The front-end server 2 selects the fragment server 3 that stores the generated fragments based on the disk selection index file 213, and selects each fragment according to the selection result based on, for example, the KVS API. Send to 3.

  FIG. 3 is a conceptual diagram showing an example of the relationship between the object OB and the disk 31 of the fragment server 3. In FIG. 3, the division from the object OB into a plurality of strides ST is omitted.

  The object OB is divided and encoded using an erasure code. As a result of encoding, fragments of k information symbol parts IS and m parity symbol parts PS are generated.

  Individual fragments are stored in the disks 31 of k + m different fragment servers 3. The data stored in this way can restore the original object OB even if up to m arbitrary fragments are lost.

  FIG. 3 illustrates the case where k = 6 and m = 3. In this case, each fragment is stored in the disks 31 of nine or more different fragment servers 3. Any loss of three fragments, for example, failure of three fragment servers 3 is allowed.

  The increase rate R (%) of the storage capacity after erasure coding is obtained by R = (m / k) × 100. In the example of FIG. 3, the increase rate R of the storage capacity is 50%.

  FIG. 4 is a diagram illustrating the disk selection index file 213.

  The disk selection index file 213 indicates a set of fragment servers 3 that are fragment storage destinations. Each disk 31 of each fragment server 3 is given a unique ID for identifying each disk 31 (hereinafter referred to as a disk ID). A set of fragment servers 3 serving as a fragment storage destination is represented by a set of disk IDs, and a unique ID (hereinafter referred to as a disk set ID) is assigned to each set.

  The disk selection index file 213 is calculated in advance when the fragment server 3 is added or deleted. Each front-end server 2 holds a disk selection index file 213.

  For example, the appearance probability of the disk ID in the disk selection index file 213 is proportional to the data storage capacity of each fragment server 3. For example, when the data storage capacity of the fragment server 3 with disk ID = x is 1 TB and the data storage capacity of the fragment server 3 with disk ID = y is 2 TB, the appearance probability of y is twice that of x. In the disk selection index file 213, each disk ID is randomly distributed while satisfying the appearance probability. However, since each fragment is always stored in a different disk, different disk IDs always appear in the same column, and two or more same disk IDs are not included in the same column.

  When referring to the disk selection index file 213 based on a certain disk set ID (z), the set value of the z column of the disk selection index file 213 is returned. This set is called a disk set. As described above, a disk set includes a plurality of disk IDs, but disk IDs do not overlap on one disk set. For example, the storage destination of the disk set ID = 3 is [0, 3, 8, 6, 7, 4,.

  The front-end server 2 determines a disk set ID for data to be stored (specifically, a set of fragments generated from a certain stride), and refers to the disk selection index file 213 based on the determined disk set ID. Then, the data is stored according to the disk ID corresponding to the disk set ID. Thereby, each fragment can always be stored in a different fragment server 3. The appearance probability of the disk ID is proportional to the data storage capacity of the fragment server 3. For this reason, even if there are fragment servers 3 having different data storage capacities, the fragments are stored in an appropriate distribution according to the data storage capacities of the fragment servers 3.

  FIG. 5 is a diagram illustrating the first to third methods of the deduplication method. It is assumed that data contents with the same reference numerals are the same.

  Object O1 and object O2 have the same data content A at the beginning and data content B in the middle, but the data content at the end differs between C and Z. Object O3 differs from object O2 in that data content Y is inserted at the beginning, X is inserted between data content A and data content B, and data content Z is deleted.

  For example, the first to third methods can be considered for deduplication in the object storage.

  The first method is deduplication in units of objects (for example, files). However, in the first method, for example, deduplication cannot be performed with only one bit difference between two objects, and deduplication efficiency is not high.

  In FIG. 5, the object O1 and the object O2 include overlapping data contents A and B, but do not match because they include data contents C and Z that do not overlap each other. The objects O2 and O3 include overlapping data contents A and B, but the object O2 includes data contents Z that does not include the object O3, and the object O3 includes data contents Y and X that do not include the object O2. , Disagreement. Therefore, in the first method, deduplication is not executed between the objects O1 to O3.

  The second method is a method in which an object is divided by a fixed length for performing deduplication, and deduplication is performed in units of the fixed length (hereinafter referred to as deduplication by a fixed position). In the second method, it is possible to eliminate duplication of fixed length units appearing at the same position in a plurality of objects. However, when the same data content is at different positions of a plurality of objects due to insertion or the like, the duplication of the same data content cannot be eliminated. For example, in the second method, deduplication is not applied to the portion having the one-bit difference described above. However, if there is the same data content in a fixed length unit at the same position, deduplication of the same data content at this same position is performed. Is done. In the second method of FIG. 5, the data content A and the data content B between the object O1 and the object O2 can be de-duplicated because their positions and values match. The data contents A and B are the same value between the object O2 and the object O3, but the positions are different. For this reason, the data contents are inconsistent in fixed length units between the object O2 and the object O3, and deduplication is not performed. The second method can perform more deduplication than the first method. However, in the second method, for example, when 1 byte data is inserted at the head of a certain file, deduplication is not performed.

  The third method is a method of performing deduplication at a free position without fixing the detection position of the overlapping portion. (Hereinafter referred to as de-duplication by free positions). In the third method, even if the positions in the object are different, overlapping portions can be eliminated if the data contents are the same in a fixed length unit. In the third method of FIG. 5, deduplication of the data content A and the data content B is possible between the object O1 and the object O2. It is possible to deduplicate the data contents A and B that match at different positions between the object O2 and the object O3. In the third method, for example, even when 1-byte data is inserted as described above, only the 1-byte data is not subject to deduplication, and the remaining portion is subject to deduplication, resulting in high deduplication. An effect is obtained.

  In the third method, depending on the fixed length, which is a unit for determining whether or not the objects match, the data is about 15% to 20% compared to the first method, which is deduplication for the entire file. The amount can be reduced.

  In the information processing system 1 according to the present embodiment, a first method for performing deduplication in units of objects, a second method for performing deduplication when the same data content is arranged at the same position in an object, Even if the positions in the object are the same or different, the third method of deduplicating the same data content with a fixed length can be used in combination.

  Hereinafter, a fixed length (duplication elimination unit) for determining whether or not they match in order to eliminate duplication will be referred to as stride length. In FIG. 5, the byte lengths of the data contents A, B, C, and Z are stride lengths.

  FIG. 6 is a diagram illustrating a comparison result between normal hash calculation and rolling hash calculation.

  In the present embodiment, for example, a method called rolling hash is used for calculating the fingerprint. A rolling hash can reduce the amount of calculation compared with a normal hash calculation. However, instead of the rolling hash, another method that can determine whether or not the data contents match may be used.

  When deduplication of fixed positions is performed using the normal hash function H, the fingerprint for each stride may be calculated. For example, the character string of the stride is passed to the hash function H as an argument to be calculated. The amount of calculation in this case is O (n), where n is the length of the character string (n is an integer).

However, when performing deduplication at a free position with the normal hash function H, the character string length n is all searched because the search is performed while shifting the position of the character string to be deduplicated, for example, by 1 byte (one character) at a time. The calculation amount is O (n ² ) when the calculation is performed with the hash function H at the position.

  On the other hand, the rolling hash function Ru is a function that can calculate, for example, a value obtained by adding 1 byte and a value obtained by deleting 1 byte with respect to a hash value (fingerprint) for a certain length L. By using this rolling hash function Ru, it is possible to reduce the amount of calculation of the hash value per time after the calculation of the hash value of the first character string to O (1).

  For example, FIG. 6 illustrates a case where the fingerprint of a character string of L = 5 is calculated for a character string “ABCDEFGH...” Having a character string length n.

  In FIG. 6, St is an internal state and Ct is a hash value (t is an integer). The normal hash function H and the rolling hash function Ru each output a combination of (St, Ct) as a hash calculation result.

When the normal hash function H is used, the calculation amount for the first character string “ABCDE” is O (n). Since the same calculation amount, that is, O (n) is required for all combinations of character strings obtained by shifting by one character, the calculation amount of O (n ² ) is approximately required as a whole.

  On the other hand, when the rolling hash function Ru is used, since the normal hash function H is used for the first character string “ABCDE”, the calculation amount is O (n). However, for the next character string “BCDEF”, for the rolling hash function Ru, the internal state S0 obtained by the first calculation and the argument (− “A” for excluding A from the first character string) ") And an argument (+" F ") for adding F can be given by calculation O (1). Since the calculation amount of O (1) is similarly applied to the remaining character strings, the overall calculation is approximately O (n), and the calculation amount is greatly reduced as compared with the case where only the hash function H is used. be able to.

  The overall calculation amount shown above is a convergence value when the length n of the character string is sufficiently longer than the length of the character string L.

  Hereinafter, the fragment storing process will be specifically described with reference to FIGS. 7 and 8.

  FIG. 7 is a flowchart illustrating fragment storage processing.

  FIG. 8 is a conceptual diagram illustrating fragment storage processing. The numbers given in FIG. 8 correspond to the numbers given to the steps in FIG.

  First, the user U or the application of the terminal 11 gives an identifier to each object to be stored. This is called an object ID. The object ID has a role of a file name in a conventional file system, for example.

  In step S <b> 701, when the front-end server 2 receives a request to save the object OB from the terminal 11 of the user U or the application of the terminal 11, the front-end server 2 stores the object OB in the object storage unit 211 via the transmission / reception unit 22.

  When the object OB received via the transmission / reception unit 22 is directly stored in the memory 24, the object OB does not have to be stored in the object storage unit 211.

  In step S702, the object dividing unit 201 reads the object OB into the memory 24 and divides it into strides ST. At this time, if the size of the object OB is not an integral multiple of the size of the stride ST, the object OB is padded to a size equal to the integral multiple of the stride ST. In the example of FIG. 8 (S702), the object dividing unit 201 performs padding so that the size of the object OB is three times the size of the stride, and is divided into strides A, B, and C.

  In step S703, the erasure code calculation unit 202 encodes each of the strides A, B, and C using an erasure code to generate respective fragments included in the information symbol part IS and the parity symbol part PS. In the example of FIG. 8 (S703), it is assumed that the parameters of the erasure code are k = 4 and m = 2. As a result of the erasure code calculation unit 202 encoding the stride A, four fragments A1, A2, A3 and A4 are generated as the information symbol part IS, and two fragments A5 and A6 are generated as the parity symbol part PS. Also for the strides B and C, four fragments are generated as the information symbol part IS, and two fragments are generated as the parity symbol part PS.

  In step S <b> 704, the fingerprint calculation unit 203 calculates a fingerprint for each stride A, B, and C using a rolling hash. The calculated fingerprint is stored in the fingerprint set 212. In the example of FIG. 8 (S704), as a result of the front end server 2 calculating the fingerprint for each stride A, B, and C using the rolling hash, the fingerprints FP_A, FP_B, and FP_C are generated.

  In step S705, the control unit 25 refers to the disk selection index file 213 using the fingerprints FP_A, FP_B, and FP_C for the strides A, B, and C, and determines the fragment server 3 that is the storage destination of each fragment. Since the storage destination fragment server 3 is determined by the fingerprints FP_A, FP_B, and FP_C for each of the strides A, B, and C, the stride ST having the same contents is always stored in the same fragment server 3. In the example of FIG. 8 (S705), the disk set ID = 2 is determined from the value of FP_A, and by referring to the disk selection index file 213 shown in FIG. 6, 1, 0, 9, 3, 7]. Each element of the array [6, 1, 0, 9, 3, 7] indicates the number of the storage destination fragment server 3 corresponding to the fragments A1 to A6. Similarly, for the strides B and C, the storage destination of the fragment is determined.

  In step S <b> 706, the control unit 25 transmits each fragment to the storage destination fragment server 3 through the transmission / reception unit 22. In the example of FIG. 8 (S706), each fragment is stored in the fragment server 3 in accordance with the contents of the disk selection index file 213 referred to in step S705. For example, for fragments A1 to A6 belonging to stride A, fragment A1 is fragment server K6, fragment A2 is fragment server K1, fragment A3 is fragment server K0, fragment A4 is fragment server K9, and fragment A5 is fragment. The fragment A6 is stored in the fragment server K7 in the server K3. Similarly, each of the strides B and C is stored in the storage destination fragment server 3.

  When each fragment is stored, metadata is stored together with each fragment. In the present embodiment, as one example of metadata, a fingerprint for one stride starting from a fragment to be stored and an internal state obtained by calculation using a rolling hash are stored (this is a roll sum state). (Roll sum state)). Also, a flag indicating whether or not deduplication processing has been performed is stored (this is referred to as a reference counter value). For example, the initial value of the reference counter value is 1, and the numerical value may increase each time it is referred to for deduplication. Details of the metadata will be described later in Tables 1 to 5.

  The metadata attached to each fragment will be described below using Tables 1 to 5.

Table 1 is a table illustrating metadata used in this embodiment.

  When the fragment is stored in the fragment server 3, various accompanying information such as configuration information of the entire object OB and configuration information of each stride ST is required. This accompanying information is called metadata. The metadata is stored in the fragment server 3 when the object OB is stored.

  Each metadata exemplified in Table 1 has a key / value structure, for example. The key-value configuration is a configuration in which data is stored as a combination of key information and value, and the value can be read by specifying the key information.

  Each metadata value is given key information for referring to the metadata, but the same key may be used for different metadata values. Therefore, a key prefix is assigned to each metadata, and a value obtained by combining this key prefix and key information is used as an actual key at the time of storage.

  The “replicatable?” Column in Table 1 indicates whether or not the metadata is copied and stored. Metadata about the fragment itself is not duplicated, and other metadata is duplicated.

  The outline of each key and metadata is as follows.

  In the present embodiment, an internal identifier is given to the object OB in addition to the object ID. This identifier is called a counter. For example, the counter is a monotonically increasing value of 64 bits with no sign. In the present embodiment, the counter includes, for example, first information, second information, and a front-end server ID.

  The first information indicates the second part at the first time when the corresponding object OB is internally allocated.

  The second information is a value obtained by dividing a time value representing the difference between the first time and the second time in microseconds by a predetermined value (for example, 25).

  The front end server ID is a unique identifier assigned to each front end server 2. By including this front-end server ID in the counter, for example, even if the counter is assigned by different front-end servers 2 at the same time, it is guaranteed that the value of this counter is different.

  Even when the application of the user U or the terminal 11 gives the same object ID to different objects and saves them, it is guaranteed that the counter values are different.

  In the present embodiment, it is assumed that the time managed by each front-end server 2 is synchronized. In this case, it is possible to compare which of the plurality of objects OB or the plurality of metadata related to the object OB is new by comparing the counters.

  The object meta counter is a list of counters of objects having object IDs used for key information, and the largest counter is the latest object. The object metadata is a set of object metadata, and an example is shown in Table 2 described later. Stride metadata is a set of metadata related to each stride, and an example is shown in Table 3 to be described later. The stride ID is a number indicating the number of stride in the object. Fragment metadata is metadata relating to each fragment, and an example is shown in Table 4 described later. The fragment position information is metadata for specifying the entity of the fragment by the fragment position, and is determined by a counter, an external stride ID (described later), and a fragment ID. The reference count value is the number of fragment references, and has a value of 2 or more when deduplication is performed. The error correction code is a code used for error detection such as a cyclic redundancy check (CRC). The pivot is an assumed overlapping position searched on the drive side, and is expressed by three sets of internal stride ID, offset, and pivot length. Details of the pivot will be described later.

  A set of fragment IDs having the same error correction code is classified and stored in two types of sets, a set before comparison (NC: not compared) and a set after comparison (CC: compared). When saved by the front-end server 2, it is saved as a set before comparison.

Table 2 is a table illustrating the contents of the object metadata.

Table 3 is a table illustrating the content of stride metadata.

Table 4 is a table illustrating the contents of fragment metadata.

  As for the storage destination of each metadata, it is necessary to obtain the disk set ID when the metadata is duplicated, and it is necessary to obtain the disk ID when the metadata is not duplicated.

Table 5 is a table illustrating the relationship between the metadata and the storage destination of the metadata.

  HR (x) in Table 5 is a remainder value obtained by dividing the value obtained by the hash function H (argument x) by the number of columns of the disk selection index file 213. When the disk set ID is obtained, the metadata is stored in the fragment server 3 corresponding to the fragment ID-th disk ID from the top of the disk set.

  In order to clarify the relationship between metadata, the relationship between each metadata will be described by taking as an example a procedure when the terminal 11 used by the user U reads an object.

  FIG. 9 is a flowchart illustrating a procedure for reading an object with object ID = o1.

  In step S901, the front-end server 2 reads out the object counter stored in the fragment server 3 using o1 that is the object ID as key information. The disk set ID of the storage destination of the object counter is obtained by the hash function HR (argument o1).

  In step S902, the front-end server 2 obtains the maximum counter from the obtained object counter (this value is c1), and reads the object metadata using c1 as a key. The disk set ID of the object metadata storage destination is obtained by the hash function HR (argument c1).

  In step S903, the front-end server 2 obtains the root stride disk set ID (disk set ID in which the first stride metadata is stored, see Table 2) included in the obtained object metadata, and the counter The stride metadata is read using c1 as a key.

  Next, the front-end server 2 reads the fragment in the stride.

  In step S904, the front-end server 2 obtains, for example, the i-th fragment metadata using the same disk set ID as the stride metadata.

  In step S905, the front-end server 2 obtains the fragment ID included in the fragment metadata (see Table 4), and reads the fragment position information from the same disk ID as the fragment metadata. Then, the front end server 2 reads the content of the fragment based on the fragment position information.

  In step S906, after reading all the fragments, the front-end server 2 obtains the next stride disk set ID (disc set ID including the next stride, see Table 3) included in the stride metadata, Read stride metadata for.

  In step S907, if there is a next stride, the front-end server 2 returns to step S904 and repeats the fragment reading process. When the last stride is read, the front-end server 2 ends the object reading process.

  FIG. 10 is a diagram illustrating a specific example of the duplicate fragment elimination process.

  As described above, the front-end server 2 calculates the same fingerprint for the same stride, and stores the fragments divided from the same stride in the same fragment server 3. In addition, there is a high possibility that fragments having the same error correction code have the same contents. The fragment server 3 periodically compares fragments having the same error correction code included in the NC (set of fragment IDs before comparison for deduplication, see Table 1), and fragments having the same error correction code are the same. If it is the content of, deduplication is performed. If the fragments having the same error correction code do not have the same content, the fragment server 3 changes the fragment ID of the compared fragment from NC to CC (set of fragment IDs after comparison for deduplication, see Table 1). And move.

  When performing deduplication processing, the deduplication processing unit 301 performs the following first to fourth deduplication processing. It should be noted that a fragment left by deduplication is denoted as frag_o, and a fragment to be deleted by deduplication is denoted as frag_n.

  The first process is a process of changing the fragment ID of the fragment frag_n in the fragment metadata to the fragment ID of the fragment frag_o.

  The second process is a process of incrementing the reference count value of the fragment frag_o.

  The third process is a process of deleting the fragment ID of the fragment frag_n from the set before the comparison of the fragment IDs having the same error correction code.

  The fourth process is a process for deleting the fragment frag_n.

  The metadata 1001 is metadata of the fragment frag_o before deduplication processing. The object ID of the fragment frag_o in the metadata 1001 is oid_o, the stride ID is si_o, and the fragment ID is fid_o.

  The metadata 1002 is metadata of the fragment frag_n before deduplication processing. The object ID of the fragment frag_n in the metadata 1002 is oid_n, the stride ID is si_n, and the fragment ID is fid_n.

  Assume that the fragment frag_n and the fragment frag_o are the same when saved by the front-end server 2. In this case, the error correction code crc_o of the fragment frag_o and the error correction code crc_n of the fragment frag_n have the same value. Accordingly, the deduplication processing unit 301 determines the fragment ID fid_o referred to using (CC, crc_o) as a key and the fragment ID fid_n referred to using (NC, crc_o) as a key for comparison.

  Next, the deduplication processing unit 301 refers to (BK, fid_o) and (BK, fid_n), acquires fragments frag_o and frag_n as the entity of the fragment indicated by each of the fragment IDs fid_o and fid_n, and performs deduplication processing. Execute.

  Metadata 1003 is frag_o metadata after deduplication processing. Metadata 1004 is metadata of frag_n after deduplication processing. Since the fragment frag_o and the fragment frag_n match in the metadata 1004, the deduplication processing unit 301 deletes the fragment frag_n and replaces the fragment IDfid_n with the fragment IDfid_o in the fragment metadata of frag_n. That is, the de-duplication processing unit 301 changes the destination indicating the entity of the fragment frag_n to the fragment frag_o.

  Further, the deduplication processing unit 301 changes the reference count value of the fragment fid_o to 2 in the metadata 1003, and records that deduplication has been performed in the metadata 1003.

  By the above process, deduplication at a fixed position can be realized. Next, a method for deduplication at a free position will be described.

  FIG. 11 is a diagram illustrating a fingerprint set.

  The front-end server 2 calculates the first half fingerprint and the second half fingerprint obtained by dividing the first fragment of the stride ST by 1/2. The front-end server 2 stores two types of fingerprints of the first half and the second half included in the first fragment, and the stride ST fingerprint. This set is called a fingerprint set.

  In the example of FIG. 11, the fingerprint set 212 held by the front-end server 2 for the stride A includes the fingerprint FP_a1 of the first half of the fragment A1, the fingerprint FP_a2 of the second half of the fragment A1, and the fingerprint FP_A of the entire stride A. Is included. The same applies to stride B and stride C. Hereinafter, a fingerprint that is half the fragment length is referred to as a ½ fingerprint.

  FIG. 12 is a diagram illustrating a fingerprint set storage format and search method in the front-end server 2.

  The front-end server 2 transmits the fingerprint set 212 to the fragment server 3 when the storage amount of the fingerprint set 212 reaches a certain amount. The fragment server 3 stores the received fingerprint set 212 in the storage unit 31. The contents of the fingerprint set 212 and the fingerprint set 311 may be compressed and stored by using, for example, a bloom filter. When a Bloom filter is used, the contents of the fingerprint set are stored together in one Bloom filter.

  A Bloom filter is a probabilistic data structure represented by a bit array, and is useful in determining whether an element is included in a member of a set. For example, when a Bloom filter is applied to a character string search, true is returned when a certain character string V is included in a predetermined character string set W, and false when the character string V is not included in the set W. return it.

  As a specific calculation method of the Bloom filter, first, all elements of a plurality of character string sets W are converted into one bit array using an arbitrarily defined hash function. Next, a hash function is similarly applied to the character string V to be compared to obtain a bit array. Then, by comparing the obtained bit array of the character string V and the bit array of the set W in bit units, it is determined whether or not the character string V to be compared is included in the character string set W. Specifically, if at least one 0 exists in the bit array of the set W corresponding to the position where the bit array of the character string V is 1, it is determined that the character string V is not included in the character string W. Is done.

  In the present embodiment, the fingerprint calculation unit 203 calculates each fingerprint value shown in FIG. Each fingerprint value is a value obtained by connecting fixed values arbitrarily determined according to the type of fingerprint, and is obtained by calculating each hash value. A counter is assigned to the obtained Bloom filter, and the Bloom filter is stored with the counter as key information (corresponding to FS in Table 1).

  In order to facilitate the search for the fingerprint set FPS, a fingerprint set list FPSL that can store a set of counters used in the fingerprint set FPS is generated. When the fingerprint set list FPSL is generated, a counter is assigned, and the fingerprint set list FPSL is stored using the counter as key information (corresponding to FL in Table 1). If the counter of the fingerprint set list FPSL reaches a certain value or more, a new fingerprint set list is generated.

  In addition, a fingerprint set list head FPSLH is generated to hold a set of counters used in the fingerprint set list FPSL. The key information of the fingerprint set list head FPSLH is shared as a fixed value throughout the system.

  As shown in FIG. 12, the fingerprint set list FPSL and the disk set ID of the fingerprint set FPS storage destination are obtained by a hash function HR, for example, using a counter as an argument. The storage set disk set ID of the fingerprint set list head FPSLH is a fixed value shared by the entire system.

  In order to search for the fingerprint set FPS, the fingerprint set list head FPSLH is first read to obtain a counter list of the fingerprint set list FPSL. From this counter, the fingerprint set list FPSL is read to obtain a list of fingerprint set FPS counters. From this counter, the fingerprint set FPS can be obtained in order.

  Since each fingerprint set FPS is provided with a counter, in the present embodiment, the generation order among a plurality of fingerprint sets FPS can be determined from the value of the counter.

  The processing contents of the deduplication processing at the free position will be described below.

  In the deduplication process at the free position, the deduplication processing unit 301 executes the following first to sixth processes.

  The first process is a process for calculating in advance a first partial hash value for the first part of the first data to be stored. In addition, the first hash value corresponding to the duplication detection range starting from the first portion in the first data is calculated in advance.

  The second process is a process of calculating a second partial hash value for the second part of the second data stored after the first data.

  The third process is a process for determining whether the first partial hash value matches the second partial hash value.

  In the fourth process, when the first partial hash value and the second partial hash value match in the third process, the fourth process corresponds to the duplication detection range that starts from the first part in the first data. This is a process of reading the hash value of 1 and calculating the second hash value corresponding to the duplication detection range starting from the second portion in the second data.

  The fifth process is a process for determining whether or not the first hash value and the second hash value match.

  In the sixth process, when the first hash value and the second hash value match in the fifth process, the duplication detection range starting from the second portion in the second data is the first in the first data. This is a process for storing data in a state where duplication is eliminated because it overlaps with the overlap detection range starting from part 1.

  Hereinafter, the deduplication processing at the free position will be described more specifically with reference to FIGS.

  FIG. 13 is a flowchart illustrating the deduplication processing at a free position.

  In step S <b> 1301, the front end server 2 transmits the fingerprint set 212 to the fragment server 3. The transmission timing may be arbitrary. For example, the transmission may be performed when the size of the fingerprint set 212 reaches a certain amount. Further, the same fingerprint set may be transmitted to all the fragment servers 3.

  In step S <b> 1302, the fragment server 3 receives the fingerprint set from the front end server 2 and stores it in the storage unit 31.

  In step S1303, the deduplication processing unit 301 of the fragment server 3 reads the content of the fingerprint set 311 stored in the storage unit 31 and temporarily stores it.

  The de-duplication processing unit 301 calculates a ½ fingerprint while shifting one byte at a time from the beginning of the fragment stored in the fragment storage unit 312 by ½ of the fragment. The rolling hash shown in FIG. 6 may be applied to the 1/2 fingerprint calculation. When there are a plurality of front-end servers 2, the fragments stored in the fragment storage unit 312 may be received from any front-end server 2.

  In step S1304, the deduplication processing unit 301 compares the fingerprint calculated in step S1303 with the ½ fingerprint of the first half or the latter half of the fragment stored in the read fingerprint set. That is, the duplicate elimination processing unit 301 performs a first duplicate search. Multiple fingerprint sets may be read.

  If the ½ fingerprint does not match, the process returns to step S1304, and the ½ fingerprint is calculated and compared while sequentially shifting by 1 byte until the ½ fingerprint matches.

  If the ½ fingerprints match, in step S1305, the de-duplication processing unit 301 sets the first fragment and the last fragment among the fragments for one stride from the matching position of the ½ fingerprint. Read from other fragment servers and calculate the fingerprint for one stride from the matching position. Details of the fragment reading process will be described later with reference to FIG.

  For example, the rolling hash shown in FIG. 6 may be applied to calculate the fingerprint for one stride. In a rolling hash, when calculating a fingerprint for one stride from a position shifted by n bytes from the beginning of a fragment, the internal state of the rolling hash calculation for one stride starting from that fragment and the stride to be calculated are calculated. It is only necessary to know the contents of two fragments including the beginning and the end, and the fragment between them is not necessary for the calculation. Specifically, the deduplication processing unit 301 sets the roll sum state stored as fragment metadata as an initial value, and calculates a fingerprint with a shift of 1 byte as shown in FIG. By repeating this n times, the fingerprint for one stride that is n bytes away from the beginning of the fragment can be calculated. When shifting one byte at a time, one byte to be added and one byte to be added can be known from the two fragments including the head and the end.

  In step S1306, the deduplication processing unit 301 compares the fingerprint of one stride calculated in step S1305 with the fingerprint of one stride stored in the read fingerprint set. That is, the duplicate elimination processing unit 301 performs a second duplicate search.

  When the fingerprints in the stride unit match, the matching location is called a pivot. As described above, the pivot is represented by a value including the internal stride ID, the offset, and the pivot length. The offset indicates how many bytes of the stride the matching position is. The pivot length indicates the length when the pivot is continuous. The internal stride ID is a number indicating to which stride it belongs when the object is subdivided by the pivot. Details of the internal stride will be described later.

  In step S 1307, the duplicate elimination processing unit 301 stores the obtained pivot in the duplicate position information 313.

  If the fingerprints in units of strides do not match, the process returns to step S1304.

  In step S <b> 1308, the fragment server 3 transmits to the front-end server 2 information about the pivot stored in the overlapping position information 313 having a certain amount. The transmission timing may be a timing for storing the pivot in the overlap position information 313, for example.

  In step S1309, when the front-end server 2 receives the list of pivots, the front-end server 2 rearranges the strides so that the pivot position is at the head of the start stride. Details of the rearrangement process will be described later with reference to FIGS.

  FIG. 14 is a diagram illustrating a first fragment reading process.

  FIG. 15 is a diagram illustrating a second fragment reading process.

  As described above, when the ½ fingerprint calculated by the deduplication processing unit 301 matches the ½ fingerprint stored in the fingerprint set, the deduplication processing unit 301 determines that one stride from the matching position. Among the minute fragments, the first and last fragments are read out, and the fingerprint for one stride is calculated from the coincidence position based on the read first and last fragments. Then, the de-duplication processing unit 301 compares the calculation result with the fingerprint of one stride stored in the fingerprint set.

  Here, out of the ½ fingerprints stored in the fingerprint set, whether it matches the ½ fingerprint of the first half of the fragment or the ½ fingerprint of the second half of the fragment, The fragment read when calculating the fingerprint for one stride may be different.

  FIG. 14 exemplifies the reading process in the case where the first fingerprint matches the ½ fingerprint. The upper part of FIG. 14 represents the fragments stored in the fragment server 3, and the lower part of FIG. 14 represents the fingerprints (corresponding to FIG. 11) stored in the fingerprint set.

  In FIG. 14, the fingerprint of the ½ fragment 1401 that is being searched by the deduplication processing unit 301 matches FP_a1 (the ½ fingerprint of the first half of the fragment A1). Therefore, the deduplication processing unit 301 calculates a fingerprint for one stride from the position corresponding to the head of the fragment A1 in order to check the next match with FP_A (stride A fingerprint). At this time, a fragment (or portion) including the head of a stride having a length of one stride from a position corresponding to the head of the fragment A1 and a fragment (or portion) including the end may be sufficient. Since the fragment including the head of the stride is the same as the head of the ½ fragment 1401 that has already been calculated by the deduplication processing unit 301, it is not necessary to read the fragment. Accordingly, the deduplication processing unit 301 reads the fragment 1402 including the end of the stride from another fragment server.

  On the other hand, in FIG. 15, the fingerprint of the ½ fragment 1403 being searched by the deduplication processing unit 301 matches FP_a2 (the ½ fingerprint of the second half of the fragment A1). Therefore, the deduplication processing unit 301 calculates a fingerprint for one stride from the position corresponding to the head of the fragment A1 in order to check the next match with FP_A (stride A fingerprint). At this time, a fragment (or portion) including the head of a stride having a length of one stride from a position corresponding to the head of the fragment A1 and a fragment (or portion) including the end may be sufficient. Accordingly, the de-duplication processing unit 301 reads the fragment 1404 including the head of the stride and the fragment 1405 including the end of the stride from other fragment servers.

  FIG. 16 is a diagram illustrating a first example of deduplication processing at a free position.

  FIG. 17 is a diagram illustrating a second example of deduplication processing at a free position. FIG. 17 expresses the detection of duplication in FIG. 16 in an easy-to-understand manner.

  The numbers given in FIGS. 16 and 17 correspond to the numbers given to the respective steps in FIG.

  16 and 17, the front end server P1 transmits the fingerprint set 212 to the fragment server K1. The fragment server K1 stores the received fingerprint set 212 in the storage unit 31 as the fingerprint set 311.

  In the fragment server K1, it is assumed that the fragment A3 is a search target. The deduplication processing unit 301 of the fragment server K1 calculates a rolling hash of ½ fragment length from the head for the fragment A3 in order, and holds the fingerprints of the first half or the latter half of the fingerprint set 311 held. Compare.

  When the deduplication processing unit 301 of the fragment server K1 finds the same ½ fingerprint at a position advanced by n bytes from the beginning of the fragment A3, the fragment B3 which is a fragment after one stride is read from another fragment server K0. .

  The deduplication processing unit 301 of the fragment server K1 uses the roll sum state stored in the fragment metadata of the fragment B3 and the fragment A3, and prints one stride fingerprint starting from the point advanced n bytes from the beginning of the fragment A3. Calculate and compare the calculated stride fingerprint with the stride fingerprint of the fingerprint set 311.

  If there is a portion in the stride fingerprint that matches the calculated one stride fingerprint, the deduplication processing unit 301 of the fragment server K1 adds this pivot position. Specifically, the pivot position may be the number of bytes from the fragment A1 that is the head of the stride.

  The deduplication processing unit 301 of the fragment server K1 stores the pivot position in the duplication position information 313. Thereafter, the fragment server K1 transmits a counter value specifying the object to be rearranged and the overlapping position information 313 to the front-end server P1.

  FIG. 18 is a diagram illustrating a first example of object rearrangement by the front-end server 2.

  When the front-end server 2 receives the counter value specifying the object to be rearranged and the overlapping position information 313 from the fragment server 3, the front-end server 2 refers to the disk selection index file 213 from the counter value and specifies the fragment server 3 that stores the pivot. . Further, the front-end server 2 reads the target object for rearrangement of the stride. At this time, the front-end server 2 also reads a reference count value indicating whether or not deduplication has already been performed for each fragment. The reference count value is 1 in the initial state, that is, the state where deduplication is not performed, and is incremented by 1 each time a reference for deduplication is performed.

  The front-end server 2 divides the read object into strides according to each pivot. The stride generated at this time is called the internal stride, and the actual storage unit stride is called the external stride. The length of the outer stride is always equal to the stride length, ie, an integral multiple of the fragment length, but the inner stride may be shorter than the stride length. A number given in order to the internal stride is called an internal stride ID. However, the internal stride is generated so that fragments that have already been deduplicated (reference count value of 2 or more) are not moved.

  In the example of FIG. 18, it is assumed that there is a pivot in the middle of the fragment A3. First, in the state (1-1), all the fragments are not yet symmetrical with respect to deduplication, that is, the reference count value is 1. When there is a possibility that an overlap portion exists from the pivot position of the fragment A3 by one stride, that is, in the middle of the fragment B3, as shown in the state (1-2), a stride boundary is created at the pivot position. Internal strides is1 to is4 are generated.

  When a plurality of internal strides are somewhat shorter than the stride length, they can be combined into one external stride. Such a stride is called a collective stride. In the state (1-3), a collective stride including the internal stride is1 and the internal stride is4 is illustrated. In the collective stride in the state (1-3), the fragment C4 is padded. In the collective stride, it is evaluated whether other internal strides can be inserted into the remaining size after the internal strides are combined. If the sum of the lengths of the internal stride is1, the internal stride is4, and the header required by combining the internal stride is1 and the internal stride is4 with the collective stride is shorter than the stride length, Padding is performed on the stride os1.

  The front-end server 2 calculates erasure codes for the external strides os1 to os3, and stores the external strides os1 to os3 by dividing them into fragment lengths. For this reason, when reading the object, the external strides os1 to os3 are sequentially read.

  However, when reading an object, it is necessary to arrange the internal strides is1 to is4 included in the external strides os1 to os3 in the order of the internal stride IDs. Therefore, when the positional relationship of the internal strides is1 to is4 is changed in order to generate the aggregate stride, the read efficiency increases as the new positions of the internal strides is1 to is4 are closer to the original positions of the internal strides is1 to is4. Will be better.

  FIG. 19 is a diagram illustrating a second example of object rearrangement by the front-end server 2.

  In the state (2-1) of FIG. 19, deduplication has already been applied to the stride composed of the fragments C1 to C4. In this case, the reference count value of the fragments C1 to C4 is 2 or more. Padding is performed such that the stride boundary is located at the pivot position. In the state (2-2), since the overlapping fragments C1 to C4 cannot be moved, padding data is inserted after the fragment B4.

  The front-end server 2 stores this rearranged stride. This saving process is similar to the case of saving the object first, and the deduplication process by each fragment server 3 is applied recursively. By the above procedure, it is possible to realize deduplication of free positions.

  In the information processing system 1 according to the present embodiment described above, each fragment server 3 shares and searches for overlapping positions. Thereby, even if the data storage amount increases, the load on the front-end server 2 does not increase. Therefore, the above-described deduplication method can be executed without reducing performance even when the storage capacity increases.

  The information processing system 1 according to the present embodiment aims to realize a low price, large capacity, and highly reliable storage, and has the following features.

  In the information processing system 1 according to the present embodiment, the increase / decrease of the storage capacity is realized by adding or deleting fragment servers, and there is no dependency on the number and configuration of the front-end servers 2. Therefore, the storage administrator only needs to manage the number of fragment servers 3 for the increase and decrease of the storage capacity, and the management is easy.

  In the information processing system 1 according to the present embodiment, the processing capacity of the entire system is improved in proportion to the number of front-end servers 2. Therefore, for example, when the performance is insufficient due to an increase in the number of storage users, the processing capacity of the entire system can be improved by simply adding the front-end server 2. The front-end server 2 is not the final storage location of data, and the number and configuration of the front-end servers 2 do not affect the number and configuration of the fragment servers 3. Therefore, the storage administrator only needs to manage the number of front-end servers 2 for performance, and management is easy.

  In the information processing system 1 according to the present embodiment, deduplication is performed in units of strides when deduplication is performed at fixed positions and deduplication is performed at free positions. That is, even after deduplication processing, the fragments generated from one stride are stored on different disks as before deduplication processing. As a result, the reliability of the erasure code is maintained even after the de-duplication processing, so that fault tolerance is ensured.

  In the information processing system 1 according to the present embodiment, the front end server 2 and the fragment server 3 are connected by a normal IP network. For this reason, for example, during the busy season, the operation of temporarily supplementing the front-end server 2 with a virtual machine on the in-house cloud is also facilitated. Further, when adding or deleting the fragment server 3, it is only necessary to maintain the disk selection index file 213, so that it is easy to obtain the capacity desired by the user. In other words, since the system configuration can be expanded or reduced as necessary, it is no longer necessary to introduce a large-scale system in advance in consideration of future increases in processing volume when introducing the system. Can be reduced.

  In the information processing system 1 according to the present embodiment, the cause of the failure can be easily analyzed by using, for example, two types of components, that is, the front-end server 2 and the fragment server 3.

  Further, data stored by the user is finally stored only in the fragment server 3. That is, since the number of components that hold data is limited to one, recovery and analysis at the time of failure can be facilitated.

  In the information processing system 1 according to the present embodiment, an erasure code is used to ensure reliability. Therefore, the data storage efficiency can be improved as compared with a technique (for example, RAID (Redundant Arrays of Inexpensive Disks) 1) that ensures reliability by duplicating data used for a large-scale storage.

  In the present embodiment, data match or mismatch is determined by comparing hash values, for example. However, data matching or mismatching may be determined by other methods as long as efficient processing is possible. For example, instead of the hash value, other information for determining whether data matches or does not match may be used.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 ... Information processing system, 2 ... Front end server, 3 ... Fragment server, 20, 30 ... Controller, 21, 31 ... Storage part, 22, 32 ... Transmission / reception part, 23, 33 ... Processor, 24, 34 ... Memory, 25 , 35 ... control unit, 201 ... object division unit, 202 ... erasure code calculation unit, 203 ... fingerprint calculation unit, 211 ... object storage unit, 212, 311 ... fingerprint set, 213 ... disk selection index file, 301 ... duplication Exclusion processing unit, 312... Fragment storage unit, 313.

Claims (9)

Comprising a first information processing device and a plurality of second information processing devices;
The first information processing apparatus includes:
A generator that generates a plurality of storage data of a storage unit size based on the data;
A first hash value corresponding to first fragment data included in the data and having a duplicate detection size larger than the storage unit size is calculated, and a first hash value included in the first fragment data is calculated. A calculation unit for calculating a second hash value corresponding to at least a part of the stored data;
A transmission unit that transmits the first storage data, the first hash value, and the second hash value to a storage destination device among the plurality of second information processing devices;
Comprising
The storage destination device is:
Receiving the first storage data, the first hash value, and the second hash value, and the second storage of the storage unit size generated by the external information processing apparatus based on the storage target data A receiver for receiving data;
Whether the second hash value corresponding to at least a part of the first stored data matches a third hash value corresponding to at least a part of the second stored data received by the receiving unit If the second hash value matches the third hash value, the first hash value corresponding to the first fragment data is the second information processing device. And whether the first hash value matches the fourth hash value corresponding to the second fragment data having the second stored data and corresponding to the second fragment data of the duplicate detection size. A processor that detects duplication of the second fragment data when the value matches,
An information processing system comprising: When the first hash value matches the fourth hash value, the processing unit transmits overlapping position information indicating a matching position in the storage target data to the external information processing apparatus,
The external information processing device generates a plurality of new fragment data for the storage target data based on the overlap position information so that an overlap portion is included in the same fragment data, and the plurality of new fragments Generating a plurality of new storage data based on the data, transmitting the plurality of new storage data to the plurality of second information processing devices;
The receiving unit receives the new saved data,
The processing unit performs deduplication of the new storage data when the new storage data matches the first storage data.
The information processing system according to claim 1. The first information processing apparatus includes:
A storage unit for storing selection information in which a reference hash value and storage destination identification information determined corresponding to the reference hash value are associated;
A control unit that determines a second information processing apparatus that is a storage destination of each of the plurality of storage data based on the first hash value and the selection information;
Further comprising
The transmission unit transmits each of the plurality of stored data to the determined second information processing apparatus;
The information processing system according to claim 1. The processing unit calculates the third hash value corresponding to at least a part of the second stored data based on a rolling hash function while shifting a calculation range in the second stored data.
The information processing system according to claim 1. The second hash value includes a hash value of the first half and a hash value of the second half of the first stored data included in the first fragment data,
The processing unit determines whether the first hash value matches the fourth hash value when the hash value of the first half part or the hash value of the second half part matches the third hash value. And when the first hash value matches the fourth hash value, the duplication with respect to the second fragment data is detected.
The information processing system according to claim 1. When the second hash value matches the third hash value, the processing unit calculates the fourth hash value corresponding to the second fragment data corresponding to the duplicate detection size starting from the matching position. Calculating and detecting duplication for the second fragment data when the first hash value matches the fourth hash value;
The information processing system according to claim 1. The generating unit generates the plurality of stored data including a plurality of divided data obtained by dividing the storage target data and error correction data corresponding to the plurality of divided data;
The information processing system according to claim 1. Corresponding to first storage data among a plurality of storage data having a storage unit size generated based on the data, and first fragment data having a duplicate detection size included in the data and larger than the storage unit size And a second hash value corresponding to at least a part of the first stored data included in the first fragment data and stored by an external information processing apparatus A receiving unit for receiving second storage data of the storage unit size generated based on data;
Whether the second hash value corresponding to at least a part of the first stored data matches a third hash value corresponding to at least a part of the second stored data received by the receiving unit If the second hash value matches the third hash value, the first hash value corresponding to the first fragment data is stored in a plurality of information processing devices. And determining whether the second hash value matches the fourth hash value corresponding to the second fragment data of the duplicate detection size, and the first hash value matches the fourth hash value. A processor for detecting duplication with respect to the second fragment data,
An information processing apparatus comprising: Computer
Corresponding to first storage data among a plurality of storage data having a storage unit size generated based on the data, and first fragment data having a duplicate detection size included in the data and larger than the storage unit size And a second hash value corresponding to at least a part of the first stored data included in the first fragment data and stored by an external information processing apparatus A receiving unit for receiving second storage data of the storage unit size generated based on data;
Whether the second hash value corresponding to at least a part of the first stored data matches a third hash value corresponding to at least a part of the second stored data received by the receiving unit If the second hash value matches the third hash value, the first hash value corresponding to the first fragment data is stored in a plurality of information processing devices. And determining whether the second hash value matches the fourth hash value corresponding to the second fragment data of the duplicate detection size, and the first hash value matches the fourth hash value. A processor for detecting duplication with respect to the second fragment data,
Program to make it function.

Images